Providing Supplemental Content

ABSTRACT

Computing devices, computer-readable media storing computer-executable instructions, methods, and systems for discovering, organizing, and presenting supplemental content are disclosed. A discovery system may be used to discover an item, such as a supplemental content presentation application, which, when launched may present supplemental content to a user while the user is consuming primary content. The supplemental content presentation application may also present a timeline that users may interact with to view different portions of the supplemental content. The timeline may also be modified by adding markers along the timeline so that when a marker is selected, a particular piece of supplemental content is presented.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of and claims the benefit ofco-pending U.S. application Ser. No. 13/771,592, filed Feb. 20, 2013,which is a continuation of U.S. patent application Ser. No. 10/630,815,which was filed on Jul. 29, 2003 and is a continuation-in-part of U.S.patent application Ser. No. 10/390,064, which was filed on Mar. 14,2003. The disclosures of each of these documents are incorporated byreference herein in their entirety.

BACKGROUND

Television viewing is no longer the static, isolated, passive pastimethat it used to be. Today, viewers have the option of using a computingdevice, such as a tablet computer, to view a webpage related to a showthey are watching, thereby keeping the viewers engaged in a particularprogram. However, there are many other webpages, and there is a demandfor new and interesting ways to keep the viewer engaged with the webpagethat is related to the particular program.

SUMMARY

Some or all of the various features described herein may facilitatediscovery, organization, and presentation of supplemental content (e.g.,second screen content, or same device companion content) on a seconduser device (e.g., a second screen device such as a tablet computer,smartphone, laptop, etc.) or a first user device (e.g., a first screendevice such as a television or video display) to complement primarycontent displayed on a first user device thereby providing a desirablesecond screen, or augmented first screen, experience.

In accordance with aspects of the disclosure, an item detection systemis provided for supplying appropriate items, such as computerapplications, Internet pages, and other interactive content, based oncontext information regarding a user's current activity. The detecteditems may be supplied to various user devices for presentation in avariety of screens. For example, the items may be presented in aninterface, e.g., a program guide, or other screens accessible through afirst screen device, such as a television, or second screen device, suchas a tablet. The item detection system, therefore, may provide a meansthrough which users may discover items related to content they areconsuming. Additional features of the item detection system with respectto how context information is obtained, how items are detected, howdetected items are arranged, how detected items are presented, and howdetected items may be downloaded and/or launched are also taught in thisdisclosure.

Further, other aspects of the disclosure include a supplemental contentpresentation application and a system for supporting said application.In an illustrative embodiment, this application may include a timelineof events relating to a program, such as a video program. The system mayprovide this timeline to said application running on a user device, suchas a tablet computer, which may present the timeline on a screen for auser to view. The timeline may be utilized to synchronize supplementalcontent with primary content so that, as the primary content ispresented to the user on the same user device or a different one,corresponding supplemental content may be presented on the user device.Users may also interact with the timeline to select points along thetimeline, which are associated with portions of the primary content, andaccess supplemental content corresponding to those points.

The system may receive and/or provide updates to the timeline from anadministrative entity and may generate instructions, includingsupplemental content, that cause the user device to modify the timelineto present the supplemental content at a point along the timeline.Modifying the timeline may include adding a marker on the timeline.Users may also modify the timeline by performing various actions thatmay cause other markers to be added to the timeline. Further, the systemmay receive data feeds from social network services and other newssources. The data feeds may include messages that the system may deliverto a user device. A user of the user device may select one of themessages, thereby causing a marker to be added to the timeline. Theselection may also create a report that is sent to the system. Based ona number of reports, the system may determine whether a marker should beadded to the timeline so that a message may be featured for other usersto consume.

Additionally, aspects of the present disclosure teach computing devices,having a processor and memory storing computer-executable instructions,and other apparatuses to perform the above steps and other steps fordiscovering items and improving a second screen experience.

Other details and features will also be described in the sections thatfollow. This summary is not intended to identify critical or essentialfeatures of the inventions claimed herein, but instead merely summarizescertain features and variations thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

Some features herein are illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements.

FIG. 1 illustrates an example communication network on which variousfeatures described herein may be used.

FIG. 2 illustrates an example computing device and softwareconfiguration that can be used to implement any of the methods, servers,entities, and computing devices described herein.

FIG. 3 illustrates a system architecture according to one or moreaspects described herein.

FIGS. 4A-4H are diagrams illustrating example user experienceinterfaces.

FIGS. 5A-5C are diagrams illustrating example embodiments of varioususer interfaces.

FIG. 6 illustrates a system architecture according to one or moreaspects described herein.

FIG. 7 is a flow diagram illustrating an example method according to oneor more aspects of the disclosure.

FIG. 8 is a diagram illustrating an example user interface with atimeline.

FIG. 9 is a diagram illustrating an example commenting user interface.

FIG. 10 is a diagram illustrating an example interaction user interface.

FIG. 11 is a diagram illustrating an example user interface.

FIG. 12 illustrates a system architecture according to one or moreaspects described herein.

FIG. 13 is a flow diagram illustrating an example method according toone or more aspects of the disclosure.

DETAILED DESCRIPTION

In the following description of various illustrative embodiments,reference is made to the accompanying drawings, which form a parthereof, and in which is shown, by way of illustration, variousembodiments in which aspects of the disclosure may be practiced. It isto be understood that other embodiments may be utilized, and structuraland functional modifications may be made, without departing from thescope of the present disclosure.

By way of introduction, the various features described herein may allowa user to discover an item, such as a supplemental content presentationapplication, and download that item to a second screen device (e.g., atablet), or interact with that content on a first screen (e.g.Television or other device presenting audio or video content). If thesupplemental content presentation application is downloaded, the secondscreen device may present supplemental content to a user while the useris consuming primary content on a first screen device (e.g., atelevision). If the supplemental content presentation application isused on the first screen device then the supplemental content may bepresented to a user while the user is consuming primary content on afirst screen device in one of many modes, in which, e.g., interactivecontent overlays video content, or interactive content is presentedbeside or around video content. A companion content experience (alsoreferred to as a second screen experience), in which supplementalcontent may be presented on a first screen device or second screendevice, may be enhanced by various features of the supplemental contentpresentation application, such as a timeline that users may interactwith and modify.

FIG. 1 illustrates an example communication network 100 on which many ofthe various features described herein, such as the requesting andretrieval of primary content, items related to the primary content, andsupplemental content may be implemented. Network 100 may be any type ofinformation distribution network, such as satellite, telephone,cellular, wireless, etc. One example may be an optical fiber network, acoaxial cable network, or a hybrid fiber/coax distribution network. Suchnetworks 100 use a series of interconnected communication links 101(e.g., coaxial cables, optical fibers, wireless, etc.) to connectmultiple premises 102 (e.g., businesses, homes, consumer dwellings,etc.) to a local office or headend 103. The local office 103, which canbe a data processing facility, may transmit downstream informationsignals onto the links 101, and each premises 102 may have a receiverused to receive and process those signals.

There may be one link 101 originating from the local office 103, and itmay be split a number of times to distribute the signal to variouspremises 102 in the vicinity (which may be many miles) of the localoffice 103. The links 101 may include components not illustrated, suchas splitters, filters, amplifiers, etc. to help convey the signalclearly, but in general each split introduces a bit of signaldegradation. Portions of the links 101 may also be implemented withfiber-optic cable, while other portions may be implemented with coaxialcable, other lines, or wireless communication paths.

The local office 103 may include an interface, such as a terminationsystem (TS) 104. More specifically, the interface 104 may be a cablemodem termination system (CMTS), which may be a computing deviceconfigured to manage communications between devices on the network oflinks 101 and backend devices such as servers 105-107 (to be discussedfurther below). The interface 104 may be as specified in a standard,such as the Data Over Cable Service Interface Specification (DOCSIS)standard, published by Cable Television Laboratories, Inc. (a.k.a.CableLabs), or it may be a similar or modified device instead. Theinterface 104 may be configured to place data on one or more downstreamfrequencies to be received by modems at the various premises 102, and toreceive upstream communications from those modems on one or moreupstream frequencies.

The local office 103 may also include one or more network interfaces108, which can permit the local office 103 to communicate with variousother external networks 109. These networks 109 may include, forexample, networks of Internet devices, telephone networks, cellulartelephone networks, fiber optic networks, local wireless networks (e.g.,WiMAX), satellite networks, and any other desired network, and thenetwork interface 108 may include the corresponding circuitry needed tocommunicate on the external networks 109, and to other devices on thenetwork such as a cellular telephone network and its corresponding cellphones.

As noted above, the local office 103 may include a variety of servers105-107 that may be configured to perform various functions. Forexample, the local office 103 may include a push notification server105. The push notification server 105 may generate push notifications todeliver data and/or commands to the various premises 102 in the network(or more specifically, to the devices in the premises 102 that areconfigured to detect such notifications). The local office 103 may alsoinclude a content server 106. The content server 106 may be one or morecomputing devices that are configured to provide content to users attheir premises. This content may be, for example, video on demandmovies, television programs, songs, text listings, etc. The contentserver 106 may include software to validate user identities andentitlements, to locate and retrieve requested content, to encrypt thecontent, and to initiate delivery (e.g., streaming) of the content tothe requesting user(s) and/or device(s).

The local office 103 may also include one or more application servers107. An application server 107 may be a computing device configured tooffer any desired service, and may run various languages and operatingsystems (e.g., servlets and JSP pages running on Tomcat/MySQL, OSX, BSD,Ubuntu, Redhat, HTML5, JavaScript, AJAX and COMET). For example, anapplication server may be responsible for collecting television programlistings information and generating a data download for electronicprogram guide listings. Another application server may be responsiblefor monitoring user viewing habits and collecting that information foruse in selecting advertisements. Yet another application server may beresponsible for formatting and inserting advertisements in a videostream being transmitted to the premises 102. Although shown separately,one of ordinary skill in the art will appreciate that the push server105, content server 106, and application server 107 may be combined.Further, here the push server 105, content server 106, and applicationserver 107 are shown generally, and it will be understood that they mayeach contain memory storing computer executable instructions to cause aprocessor to perform steps described herein and/or memory for storingdata, such as information for identifying a user or second screendevice.

An example premises 102 a, such as a home, may include an interface 120.The interface 120 can include any communication circuitry needed toallow a device to communicate on one or more links 101 with otherdevices in the network. For example, the interface 120 may include amodem 110, which may include transmitters and receivers used tocommunicate on the links 101 and with the local office 103. The modem110 may be, for example, a coaxial cable modem (for coaxial cable lines101), a fiber interface node (for fiber optic lines 101), twisted-pairtelephone modem, cellular telephone transceiver, satellite transceiver,local wi-fi router or access point, or any other desired modem device.Also, although only one modem is shown in FIG. 1, a plurality of modemsoperating in parallel may be implemented within the interface 120.Further, the interface 120 may include a gateway interface device 111.The modem 110 may be connected to, or be a part of, the gatewayinterface device 111. The gateway interface device 111 may be acomputing device that communicates with the modem(s) 110 to allow one ormore other devices in the premises 102 a, to communicate with the localoffice 103 and other devices beyond the local office 103. The gateway111 may be a set-top box (STB), digital video recorder (DVR), computerserver, or any other desired computing device. The gateway 111 may alsoinclude (not shown) local network interfaces to provide communicationsignals to requesting entities/devices in the premises 102 a, such asdisplay devices 112 (e.g., televisions), additional STBs 113, personalcomputers 114, laptop computers 115, wireless devices 116 (e.g.,wireless routers, wireless laptops, notebooks, tablets and netbooks,cordless phones (e.g., Digital Enhanced Cordless Telephone—DECT phones),mobile phones, mobile televisions, personal digital assistants (PDA),etc.), landline phones 117 (e.g. Voice over Internet Protocol—VoIPphones), and any other desired devices. Examples of the local networkinterfaces include Multimedia Over Coax Alliance (MoCA) interfaces,Ethernet interfaces, universal serial bus (USB) interfaces, wirelessinterfaces (e.g., IEEE 802.11, IEEE 802.16), analog twisted pairinterfaces, Bluetooth interfaces, and others.

FIG. 2 illustrates general hardware elements that can be used toimplement any of the various computing devices discussed herein. Thecomputing device 200 may include one or more processors 201, which mayexecute instructions of a computer program to perform any of thefeatures described herein. The instructions may be stored in any type ofcomputer-readable medium or memory, to configure the operation of theprocessor 201. For example, instructions may be stored in a read-onlymemory (ROM) 202, random access memory (RAM) 203, removable media 204,such as a Universal Serial Bus (USB) drive, compact disk (CD) or digitalversatile disk (DVD), floppy disk drive, or any other desired storagemedium. Instructions may also be stored in an attached (or internal)hard drive 205. The computing device 200 may include one or more outputdevices, such as a display 206 (e.g., an external television), and mayinclude one or more output device controllers 207, such as a videoprocessor. There may also be one or more user input devices 208, such asa remote control, keyboard, mouse, touch screen, microphone, etc. Thecomputing device 200 may also include one or more network interfaces,such as a network input/output (I/O) circuit 209 (e.g., a network card)to communicate with an external network 210. The network input/outputcircuit 209 may be a wired interface, wireless interface, or acombination of the two. In some embodiments, the network input/outputcircuit 209 may include a modem (e.g., a cable modem), and the externalnetwork 210 may include the communication links 101 discussed above, theexternal network 109, an in-home network, a provider's wireless,coaxial, fiber, or hybrid fiber/coaxial distribution system (e.g., aDOCSIS network), or any other desired network.

The FIG. 2 example is a hardware configuration. Modifications may bemade to add, remove, combine, divide, etc. components of the computingdevice 200 as desired. Additionally, the components illustrated may beimplemented using basic computing devices and components, and the samecomponents (e.g., processor 201, ROM storage 202, display 206, etc.) maybe used to implement any of the other computing devices and componentsdescribed herein. For example, the various components herein may beimplemented using computing devices having components such as aprocessor executing computer-executable instructions stored on acomputer-readable medium, as illustrated in FIG. 2. Some or all of theentities described herein may be software based, and may co-exist in acommon physical platform (e.g., a requesting entity can be a separatesoftware process and program from a dependent entity, both of which maybe executed as software on a common computing device). Additionally, thecomputing device 200 may include a supplemental content delivery manager201 a, which can perform the various methods for discovering,organizing, and presenting supplemental content described herein, as areplacement for, or augment to, any other processor 201 that thecomputing device 200 may include. That is, the supplemental contentdelivery manager 201 a may include a separate processor and/or set ofcomputer-executable instructions stored on a computer-readable mediumthat, when executed by a processor, cause the processor (or thecomputing device 200 as a whole) to perform the various methods of thepresent disclosure, such as discovering supplemental content andpresenting a timeline. The supplemental content delivery manager 201 amay also include secure memory (not shown), which can store the varioususer preferences, user profile information, and algorithms describedherein. The secure memory can be any desired type of memory, and canhave enhanced security features to help restrict access (e.g., can onlybe accessed by the supplemental content delivery manager 201 a, can beinternal to the supplemental content delivery manager 201 a, etc.).Where the supplemental content delivery manager 201 a includes aseparate set of computer-executable instructions, these instructions maybe secured such that only authorized users may be allowed to modify,augment, or delete them.

In some embodiments, the supplemental content delivery manager 201 a maybe implemented as an application specific integrated circuit (ASIC).That is, the supplemental content delivery manager 201 a may be a chipdesigned specifically for performing the various processes describedherein. Further, the ASIC may be implemented within or in communicationwith various computing devices provided herein.

One or more aspects of the disclosure may be embodied in computer-usabledata and/or computer-executable instructions, such as in one or moreprogram modules, executed by one or more computers or other devices.Generally, program modules include routines, programs, objects,components, data structures, etc. that perform particular tasks orimplement particular abstract data types when executed by a processor ina computer or other data processing device. The computer executableinstructions may be stored on one or more computer readable media suchas a hard disk, optical disk, removable storage media, solid statememory, RAM, etc. As will be appreciated by one of skill in the art, thefunctionality of the program modules may be combined or distributed asdesired in various embodiments. In addition, the functionality may beembodied in whole or in part in firmware or hardware equivalents such asintegrated circuits, field programmable gate arrays (FPGA), and thelike. Particular data structures may be used to more effectivelyimplement one or more aspects of the disclosure, and such datastructures are contemplated within the scope of computer executableinstructions and computer-usable data described herein.

FIG. 3 is a diagram showing an example system architecture 300 on whichvarious features described herein may be performed. The system 300 ofFIG. 3 depicts a local office 103, a first premises 102 a, a secondpremises 102 b, one or more content distribution networks (CDN) 310 and320, a network 330, and a second screen experience manager (e.g., acomputing device or server) 340. As shown in FIG. 1, the local office103 may connect to the first premises 102 a and second premises 102 bvia links 101. The first premises 102 a may include an interface 120(e.g., a gateway computing device, modem, digital video recorder, settop box, etc.), a first screen device 301 (e.g., a television, amonitor, a projector, etc.), and one or more second screen devices 302(e.g., a smartphone, tablet, laptop, etc.). As shown in FIG. 3, multipleusers A and B may be located at the first premises 102 a and each usermay operate a second screen device 302 while consuming primary contentvia the first screen device 301. Meanwhile, the second premises 102 bmay include an interface 120, a first screen device 301, and a secondscreen device 302 used by a user C. Primary content, such as videocontent, may be transmitted from the local office 103 to the interfaces120 of the first and second premises 102 a-b, and presented through thefirst screen devices 301. Thus, users A and B may consume primarycontent (e.g., view a television show, or a stream of a video programthat is multicast according to a schedule, or transmitted on demand, orplay back content that is locally recorded at the first screen device301 or an associated device such as at a DVR) at the premises 102 a anduser C may consume primary content at the premises 102 b. In addition topresenting primary content, the first screen device 301 may also presentsupplemental content (e.g., second screen content). Notably, the firstscreen device 301 may present a listing of items related to the primarycontent being displayed, and the user may request that one or more ofthe listed related items be downloaded to and executed on the firstscreen device 301 or a second screen device 302 at the same premises102.

Also, while consuming primary content, each user may operate arespective second screen device 302 to consume supplemental content(e.g., second screen content) related to the primary content presentedon the first screen device 301 at their premises 102. For example, userA may operate a second screen device 302, such as a smartphone, toconsume second screen content, such as a poll through which user A mayvote for a contestant shown in the primary content presented on thefirst screen device 301. The second screen content may be any data thatprovides information or content to supplement primary content, which maybe the video content (e.g., linear television program, on-demand movie,etc.) presented on a first screen device 301. For example, second screencontent may include a link to a webpage of a product shown in anadvertisement of the primary content, a video clip with bonus features,text and/or images with information about the content itself or aboutindividuals or items shown in the primary content, advertisements,coupons, questions pertaining to the primary content, etc. In someembodiments, the various second screen content may be generated fromordinary everyday consumers of the primary content, such as viewerreviews of a video program, chat room discussions, of a movie, etc. Insome embodiments, the second screen content may be from formal primarycontent sources, such as the same source that provided the primarycontent (e.g., a television company may provide both a televisionprogram as primary content, and a companion Internet page secondarycontent to accompany the display of the primary content). The appearanceof the second screen content may be generated by the second screendevice 302 using software that is previously stored, or it may bedynamically retrieved or received when it is desired, and the timing ofwhen the second screen content appears (e.g., when a particular Internetlink should appear, or when a particular image should be displayed) maybe based on triggers (e.g., Enhanced Binary Interchange Format (EBIF)triggers) or signals that are received along with, or in addition to,the primary content stream. Triggers may also be generated by othermethods such as, but not limited to, (1) by analyzing audio and or videosignals to determine a position in a program (e.g., automated contentrecognition), or (2) by explicitly accessing the media time of a videoasset. In both of these additional cases, the time within a program canbe used to compare against a list of triggers for a program in order toidentify an appropriate trigger. In any event, EBIF and/or time-basedtrigger files may be combined with contextual information to launch, oroffer for launch, supplemental content.

Referring to FIG. 3, users may consume primary content at a premises 102a (e.g., a home, business, etc.). Consuming primary content may include,for example, watching and/or listening to a television program orinternet video on a first screen device 301 (e.g., a television,computer monitor, projector, etc.). The first screen device 301 mayreceive the primary content from the interface 120, which is connectedto the local office 103 and configured to accept the primary content.FIG. 3 also illustrates some examples of second screen devices 302,namely a smartphone and a laptop computer. Each second screen device 302may be configured to store and/or execute a supplemental contentpresentation application (e.g., a computer program) through which a usermay select and consume second screen content. This application may bedownloaded from the local office 103 or another computing device 200 onthe network 330, or retrieved from a computer readable medium (e.g.,compact disc (CD), flash drive, etc.). The supplemental contentpresentation application may also be a web browser for navigating to awebsite that provides the second screen experience described herein.Although FIG. 3 shows some example second screen devices 302, many otherdevices may be used as second screen devices 302. Indeed, even anothertelevision (or the same television), similar in configuration to a firstscreen device 301, may be used as the second screen device 302. Thesecond screen device 302 may also be a specially designed device (e.g.,an enhanced television remote) for specific use in the embodimentsdisclosed herein.

Further, each of the second screen devices 302 may be configured tobi-directionally communicate via a wired and/or wireless connection withthe second screen experience manager 340 via the network 330.Specifically, the second screen devices 302 may be configured to accessthe network 330 (e.g., the Internet) to obtain second screen content andto transmit/receive signals via the network 330 to/from the secondscreen experience manager 340. For example, a second screen device 302may transmit information, such as requests for second screen content,through a wired connection, including the links 101 through which theprimary content is supplied to a first screen device 301, to the localoffice 103 which then routes the transmission to the network 330 so thatit may eventually reach the second screen experience manager 340. Thatis, the second screen device 302 may connect to the interface 120 andcommunicate with the second screen experience manager 340 over the links101 used to transmit the primary content downstream. Alternatively, asecond screen device 302 may wirelessly communicate via, for example, aWiFi connection and/or cellular backhaul, to connect to the network 330(e.g., the Internet) and ultimately to the second screen experiencemanager 340. Accordingly, although not shown, the network 330 mayinclude cell towers and/or wireless routers for communicating with thesecond screen devices 302.

Although FIG. 3 depicts the second screen experience manager 340 asbeing separate from the local office 103, in some embodiments, thesecond screen experience manager 340 may be located at the local office103. In such embodiments, the second screen devices 302 may still accessthe second screen experience manager 340 through the network 330.Further, even though the second screen experience manager 340 is shownas a single element, in some embodiments, it may include a number ofcomputing devices 200, which may include the supplemental contentdelivery manager 201 a. The second screen experience manager 340, likeother computing devices 200, may include a processor and memory storingcomputer-executable instructions that, when executed, cause theprocessor to perform a variety of functions recited herein, such asproviding a timeline.

Still referring to FIG. 3, the local office 103 may include a router305, and a database 306 for storing user information (e.g., userprofiles), primary content, second screen content, and/orcomputer-executable instructions for providing supplemental content orperforming any of the steps described herein. The router 305 of thelocal office 103 may forward requests for content from users and/or userdevices (e.g., first screen devices 301, second screen devices 302,etc.) to one or more CDNs 310 and 320 and/or the second screenexperience manager 340 that may supply the requested content and/ortimeline information. Each of the CDNs 310 and 320 may include one ormore routers 311 and 321, whose purpose is to receive requests fromusers (e.g., via their local offices) and route them to servers withinits network that may store the requested content and be able to supplyit in response to the request. A CDN 310 for a given piece of contentmight have a hierarchy of one primary source, and a plurality oflower-level servers that can store (e.g., cache) the content and respondto requests. The lower-level servers that ultimately service the requestmay be referred to as edge servers, such as one or more edge servers 312and 322. The various servers may include one or more content databases313 and 323, which store content that the respective CDN 310 and 320manages. In some embodiments, the CDNs 310 and 320 may provide the sameor similar content. In other embodiments, the CDNs 310 and 320 may offerdifferent content from one another. Also, the CDNs 310 and 320 may bemaintained/operated by the same or different content providers. Althoughonly two CDNs 310 and 320 are shown, many CDNs may be included in thesystem architecture 300 of FIG. 3.

FIG. 4A is a diagram illustrating an aspect of the present disclosure.Specifically, FIG. 4A illustrates a program guide 400, such as anelectronic program guide (EPG). This program guide 400 may be providedby the local office 103 or another computing device (e.g., the secondscreen experience manager 340). The program guide 400 provides a listingof content that may be presented on a first screen device 301 or secondscreen device 302. As shown in FIG. 4A, the listing includes a pluralityof content options 401 and may be organized in a grid where logicalchannel numbers are ordered vertically and the content on those channelsare ordered according to time in the horizontal direction. The programguide 400 may be accessed and navigated using a second screen device 302or separate remote control. For example, a user may press a “guide”button on a second screen device 302 or remote control to display theprogram guide 400 on a first screen device 301, and then pressdirectional arrow buttons (e.g., up, down, left, and/or right buttons)to navigate throughout the program guide 400. FIG. 4A shows a scenarioin which “The Office” option 401 is temporarily selected. As shown, acontent option 401 may appear shaded, highlighted, etc. when temporarilyselected so that a user can realize where he/she is navigating tothroughout the program guide 400.

When an option is selected, content information 403 related to thecontent associated with that option may be displayed in a preview frame402. In the case of FIG. 4A where “The Office” is selected, the contentinformation 403 may provide details regarding the content that may beviewed, such as a description of the content, a title of the content(e.g., “Dundee Awards”), information regarding actors/actresses in thecontent, etc. The preview frame 402 may also include a content sample404, such as an image and/or video associated with the content.

Additionally, the program guide 400 may include a discovery frame 405.The discovery frame 405 may include other items 406 related to theselected content. Examples of related items 406 may include media (e.g.,audio, video, audio/video content), links (e.g., URLs), applications (orcomputer programs), advertisements, and the like. As shown in FIG. 4A,when “The Office” is selected, the discovery frame 405 may include afile having code for rendering a video game with a video game system(e.g., Xbox) or an application for playing a video game related to “TheOffice” (“The Office Video Game”), a link to a website where merchandiserelated to “The Office” may be purchased (“The Office Shop”), anapplication for a content provider related to “The Office” (“NBC App”),and an application for presenting supplemental content related to “TheOffice” (“Watch With Me”). Notably, some items (e.g., the “Watch WithMe” item) may provide supplemental content to be consumed insynchronization with the primary content, while other items might not befor synchronous consumption with the primary content (e.g., “The OfficeVideo Game” may be played after an episode of “The Office” is over).Items appearing in the discovery frame 405 may also be related to asponsor of a program and not specifically to the content itself. Forexample, Coca Cola™ may sponsor an episode of “The Office,” and thus, aCoca Cola™ application may appear in the discovery frame 405 as relatedto “The Office.” Each of these items may be represented with an icon,which may include an image, video, text, etc. and may be selectable(e.g., may include a link). Herein, the term “items” is used to denotethe icons and/or the content, links, applications, code, etc. that theyrepresent. Additional details regarding how items 406 are chosen forpresentation in the discovery frame 405 are provided below with respectto FIGS. 6 and 7.

While a particular piece of content is selected, e.g., “The Office,” auser may choose one of the related items 406 in the discovery frame 405.The items 406 in the discovery frame 405 may automatically change as auser navigates to other options 401 within the program guide 400 so thatthe items 406 in the discovery frame 405 correspond to the selectedoption 401. That is, the discovery frame 405 may be populated andcontinuously, or intermittently, updated with items 406 identified by anitem detection system described in more detail below. In this manner, auser may discover one or more items 406 available for a particular pieceof content.

Moreover, a user may navigate the discovery frame 405 to select theitems 406 shown. While an item 406 is selected, a user may choose todownload the selected item 406 (e.g., receive and store an executablefile of the selected item 406). The item 406 may be downloaded to aninterface 120 (e.g., a set top box), a media center coupled to theinterface 120, a first screen device 301, or a second screen device 302.In some examples, the item 406 may first be downloaded to an interface120 and then forwarded to the second screen device 302 of the user whomade the selection to download and/or to other second screen devices 302in communication with the interface 120.

In some cases, one or more items 406 may have already been downloadedto, e.g., an interface 120 or media center coupled thereto. In suchcases, while an item 406 that has already been downloaded and cached isselected, a user may choose to launch the item 406 on a first or secondscreen device 301, 302. Accordingly, when an item 406 is first selected,a local computing device (e.g., the interface 120) may check todetermine whether the item 406 has been downloaded already beforerequesting it to be downloaded. To support such an embodiment, the localcomputing device may include a local cache to store a list of alreadydownloaded items and/or the items themselves. Also, where the item is alink or HTML element, selecting the item may trigger a first screendevice 301 or second screen device 302 to launch a browser to viewcontent associated with the link or HTML element.

In addition, the discovery frame 405 may include a search box (notshown) in which a user may insert keywords to find related items 406.Such a search box may be used to refine the search results in thediscovery frame 405 or to conduct a new search for all available items406. However, in some examples such a search box might not exist, andthus, a user may discover related items without searching but bynavigating the program guide 400 instead. Further, although FIG. 4Aillustrates that the items 406 may be presented within a discovery frame405, the items 406 may be arranged anywhere within the program guide400. For example, the items 406 may appear within the time/channel griditself.

FIG. 4B is a diagram illustrating another aspect of the presentdisclosure. Specifically, FIG. 4B shows an example of a contentinformation screen 410 (e.g., a program information screen). Herein,screens, such as the content information screen 410, may refer to thedata used to render an image. The content information screen 410 mayappear on a first screen device 301 or a second screen device 302 when auser chooses to view information for a particular piece of primarycontent (e.g., a television program). For example, a user may scrollover a particular program in the program guide 400 that the user isinterested in and press an “info” button on a remote control device (orsecond screen device 302) to bring up the content information screen410. As shown in FIG. 4B, the content information screen 410 may providevarious information regarding “The Office” (an example televisionprogram) that a user may be interested in. The content informationscreen 410 may include a number of menu options 411 for controllingwhich type of information is viewed in a detailed information frame 412.As shown in FIG. 4B, the “Apps” menu option 411 may be selected, andtherefore, the detailed information frame 412 may present items relatedto “The Office.” One or more related items may be shown in the detailedinformation frame 412. One of the related items may include anapplication called “Silly Jokes App” that is detected as being relatedto “The Office.” This item may have a higher rating, priority, and/ordegree of relevance, and therefore, may appear larger than other items.The detailed information frame 412 may include a “Launch App” optionand/or “Save to My Apps” option that allows a user to download andrun/store an application, such as the “Silly Jokes App.”

The detailed information frame 412 may also provide item descriptioninformation 413. The item description information 413 may include adescription of the item explaining features of the item, what platformthe item is compatible with, how much memory is needed to download theitem, how long it will take to download the item, how much the itemcosts, etc. The item description information 413 may also include arating of one or more of the items and/or a preview of one or more ofthe items (e.g., a video clip showing features of the item). Further,the item description information 413 may also include a link to a page(e.g., a webpage) including more information if the item descriptioninformation 413 does not fit within the detailed information frame 412(see the “more info” link in FIG. 4B).

FIG. 4C is a diagram illustrating another aspect of the presentdisclosure. FIG. 4C shows a favorite program screen 420 that may providea list of programs that a user identifies as a favorite. In some cases,the list of programs may also include recommendations of other programsbased on the favorites of the user. The favorite program screen 420 maybe presented on a first screen device 301 when, for example, a userpresses a “favorites” button/key on a remote control for the firstscreen device 301 or selects a “favorites” option on a program guide400. Or, the favorite program screen 420 may be presented on a secondscreen device 302 when, for example, a user runs a program installed onthe second screen device 302 that enables a user to view favoriteprograms and other information regarding content. In some embodiments,when a “favorites” option of the first screen device 301 is selected, anassociated second screen device 302 may present a listing of favoriteprogram options. For example, referring to FIG. 3, the first screendevice 301 or an interface 120 connected thereto (e.g., a set top box,digital video recorder, etc.) at the premises 102 a may transmit asignal to user A's second screen device 302 and/or user B's secondscreen device 302, so that the second screen devices 302 may present thefavorite program screen 420 of FIG. 4C. As shown in FIG. 4C, the list ofprograms may comprise one or more favorite program options 421. The usermay select one of these favorite program options to view items relatedto the selected favorite program in a discovery frame 422 of thefavorite program screen 420.

In the example of FIG. 4C, the discovery frame 422 may be similar to thedetailed information frame 412 of FIG. 4B. However, the discovery frame422 of FIG. 4C illustrates an example of item description information423 for the “Watch With Me” application. As will be described in moredetail below, the “Watch With Me” application may retrieve messages fromvarious social network services. The device (e.g., the interface 120,first screen device 301, or second screen device 302) that controls thepresentation of the discovery frame 422 may interface with the “WatchWith Me” application to extract one or more of these messages to presentthem as examples within the discovery frame 422. For example, as shownin FIG. 4C, the item description information 423 may include a messagefrom “username1” sent using TWITTER™ and a message from username2 sentusing a different social network application, such as FACEBOOK™.Accordingly, in some cases, the discovery frame 422 may serve to presentdata from the data feeds that the “Watch With Me” application receives.For example, the discovery frame 422 may include RSS feeds or TWITTER™messages transmitted from the “Watch With Me” application.

FIG. 4D is a diagram illustrating another aspect of the presentdisclosure. FIG. 4D shows a search results screen 430 for presentingresults of a user's search for programs. A user may use a feature of theprogram guide 400 to search for programs using keywords or otherinformation. In the example of FIG. 4D, a user has entered a search for“popular comedy shows,” and the search results screen 430 is presentinga list of programs that match the search criteria. The search resultsscreen 430 may include one or more result options 431 for each of theprograms detected as a result of the search. When one of the resultoptions 431 is selected, a discovery frame 432 may present items relatedto the particular result option 431. For example, in FIG. 4D, the resultoption 431 for “The Office” is selected, and therefore, the discoveryframe 432 may present items related to “The Office” program. Here, thediscovery frame 432 of FIG. 4D may be similar to the discovery frame 412of FIG. 4B since both frames present items related to “The Office”program.

Instead of performing a search as described above, in some casesprograms may be associated with a particular genre (e.g., sciencefiction, comedy, sports, movies, etc.), and a user may select aparticular genre to view items related to that genre. Notably, theprograms returned as a result of the search or selecting a genre mayinclude linear content as well as non-linear content (e.g., contentpreviously broadcasted or to be broadcasted in the future, on-demandcontent, etc.). Thus, one of skill in the art should appreciate thatitems related to non-linear content may also be discoverable using asystem in accordance with this disclosure.

FIG. 4E is a diagram illustrating yet another aspect of the presentdisclosure. FIG. 4E shows a recorded content screen 440 presenting alist of content (e.g., programs) that a user has chosen to record. Inaddition, or alternatively, the list of content may include accessibleVideo on Demand assets (e.g., content recorded by a video aggregator andavailable for viewing). The recorded content may be recorded by adigital video recorder (DVR) or other computing device 200, or offeredby Video on Demand server, so that a user can consume the content at adifferent time (e.g., a later time) than when the content is distributedby a content provider. When a user wishes to view a listing of his/herrecorded content he/she may control a first screen device 301 or secondscreen device 302 to display the recorded content screen. Whiledisplaying the recorded content screen 440, a user may select one of therecorded content options 441. When a recorded content option 441 isselected, information about the recorded content (e.g., a description ofthe recorded content, a duration of the recorded content, etc.) may bedisplayed in a recorded content information frame 442. Additionally,when a recorded content option 441 is selected, items related to therecorded content option 441 may be presented in a discovery frame 443.As shown in FIG. 4E, the selected content option 441 includes “TheOffice” program, and therefore, the discovery frame 443 may presentitems related to “The Office” in a manner similar to the discoveryframes 412 of FIGS. 4B and 422 of FIG. 4C.

FIG. 4E also illustrates another feature of the present disclosure. Thediscovery frame 443 may present items related to events within thecontent itself. For example, if one of the characters in the particularepisode of “The Office” that was recorded goes skydiving, anapplication, such as the “Skydiving App,” may be included in thediscovery frame 443. Also, if one of the characters in the particularepisode of “The Office” that was recorded goes on a date, anapplication, such as the “Online Dating App,” may be included in thediscovery frame. From these two examples, it should be understood thatvarious items that are relevant to the events within the primary contentmay exist. Further, in light of these examples, it should be realizedthat different pieces of similar primary content (e.g., differentepisodes of the same television program) may be associated withdifferent related items. To realize these features, context informationmay be embedded in the audio feed and/or video feed (e.g., in differentframes) of the primary content. Additional details regarding how itemsare chosen for presentation in the discovery frame 443 are providedbelow with respect to FIGS. 6 and 7.

FIG. 4F is a diagram illustrating another aspect of the presentdisclosure. In FIG. 4F, a piece of primary content, such as an episodeof “The Office,” may be presented on a first screen device 301 (e.g., atelevision, computer, etc.). Meanwhile, a user may execute anapplication (or computer program) to consume, via a second screen device302 (e.g., a tablet, smartphone, etc.), supplemental content related tothe primary content presented on the first screen device 301. Forexample, the second screen device 302 in FIG. 4F may run an application,referred to as the “Watch With Me App,” to view supplemental content forthe episode of “The Office” presented on the first screen device 301. Aninterface 120 connected to the first screen device 301 or the firstscreen device 301 itself may send one or more triggers 450 to the secondscreen device 302 indicating what primary content is being presented onthe first screen device 301. For example, the triggers 450 may indicatea name (e.g., “The Office”) of the primary content, a particular episodeof the particular content, and/or a particular segment or point in timeof the primary content. The triggers 450 may be sent via a wired orwireless connection. Alternatively, or additionally, the second screendevice 302 may determine what content the first screen device 301 ispresenting by analyzing audio signals 460 received from the first screendevice 301.

Using the triggers 450, audio signals 460, and/or an explicit time code,the second screen device 302 may determine which portion (or segment) ofthe content is being presented. Based on this information, the secondscreen device 302, and in particular, the “Watch With Me App” running onthe second screen device 302, may display corresponding supplementalcontent. That is, the supplemental content presented on the secondscreen device 302 may be synchronized with the content presented on thefirst screen device 301. Furthermore, the second screen device 302 maypresent related items in a discovery frame 470 that are alsosynchronized with the supplemental content and/or primary content. Forexample, as shown in FIG. 4F, a person within the primary content may berunning, and thus, the second screen device 302 may display an itemrelated to running, such as the “Running App.” While the second screendevice 302 may present items related to specific events within theprimary content, the second screen device 302 may also present itemsgenerally related to the primary content, such as the “Silly Jokes App.”The order in which these items appear within the discovery frame 470 mayvary, and may be determined based on ratings or a pay-for-placementscheme whereby creators/owners of items pay for inclusion and rankingwithin the item list as described in more detail below. Additionally, oralternatively, the order may be such that priority is given to itemsrelated to the events occurring in the primary content over moregenerally related items and vice versa. For example, in FIG. 4F, theevent of a person running may cause the “Running App” item to bepresented above (or, when the discovery frame 470 is horizontallyelongated, to the left of) the “Silly Jokes App,” which is generallyrelated to “The Office,” but not particularly relevant to the portion of“The Office” currently presented on the first screen device 301.

FIG. 4G is a diagram illustrating another aspect of the disclosure. Inparticular, FIG. 4G illustrates another manner in which to display adiscovery frame 480 including items related to primary content presentedon a first screen device 301. In FIG. 4G, the discovery frame 480 mayappear as an overlay on the first screen device 301. In other words, thediscovery frame 480 may cover a portion of the primary content presentedon the first screen device 301. The discovery frame 480 may continuouslyappear on the first screen device 301 and the items in the discoveryframe 480 may change according to changes in the primary content.Alternatively, the discovery frame 480 might only appear when relateditems are detected as corresponding to the particular piece of primarycontent. Therefore, the related items may pop-up onto the screen of thefirst screen device 301 as the primary content progresses and stay onthe screen for some predetermined period of time. In other embodiments,the discovery frame 480 might only appear when a user chooses to seerelated items. For example, the discovery frame 480 might only bedisplayed when a user controls the first screen device 301 to show thediscovery frame using a remote control. The specific items presented inthe discovery frame 480 may be detected based on triggers (not shown) asdescribed above with respect to FIG. 4F. Once discovered and presentedin the discovery frame 480, items may be selected thereby causing theitem to be launched directly on the first screen device 301 or on asecond screen device 302 in proximity to the first screen device 301.

FIG. 4H illustrates an embodiment in which an invitation to launch anitem may be presented as opposed to a discovery frame. As shown in FIG.4H, an invitation 490 to launch an item (e.g., a “Running App”) may bepresented on a first screen device 301 in response to one or moretriggers detected in the video and/or audio streams of the primarycontent being displayed on the first screen device 301. In addition, oralternatively, the invitation 490 may be presented on a second screendevice 302 in proximity to or in communication with the first screendevice 301. In any event, when the invitation is accepted (e.g., bypressing a button on a remote of the first screen device), the item maybe launched on the first screen device 301 or second screen device 302.Referring to the example in FIG. 4H, when a trigger within the primarycontent, which indicates that a person is running, is detected, aninvitation 490 to launch a related item may be displayed on the firstscreen device 301. Then, when a user selects the invitation 490, theinterface 120 may transmit a signal 495 instructing the second screendevice 302 to launch the related item. Additionally, if the secondscreen device 302 does not already have the executable file of therelated item, the signal 495 may include the executable file, or mayinclude information indicating where the second screen device 302 mayaccess the executable file.

Also, although the invitation 490 is shown in FIG. 4H, it might not beincluded within some embodiments, and instead, the item may beautomatically launched upon detection of a trigger. For example, a usermay set a user preference to automatically launch items on a particularsecond screen device 302 that are related to certain triggers when suchtriggers are detected.

FIG. 5A illustrates a webpage 500 that may be displayed on a firstscreen device 301, a second screen device 302, or other computing device200 using a web browser. In some examples, the webpage 500 may belong toa website hosted by the second screen experience manager 340.Alternatively, the host of the webpage 500 may be another computingdevice in communication with the second screen experience manager 340,or the webpage 500 may include scripting code for accessing the secondscreen experience manager 340, so that the webpage 500 may be populatedwith related items.

Notably, the webpage 500 may include a discovery frame 501. Thediscovery frame 501 may be generated by the web browser to display itemsthat are related to the webpage 500. If it is determined that there aresuch related items, then the related items may be presented in thediscovery frame 501. One manner for determining which items, if any, arerelated to the webpage 500 may be to analyze the URL of the webpage, andcompare it to a list of available items. Specifically, an item detectionsystem (discussed in further detail below) may detect items that arerelated to the webpage 500 based on the URL of the webpage 500 and itscontents, and the detected items may be presented in the discovery frame501 on the webpage 500. For example, if the webpage 500 is a webpage fora television program, such as “The Office,” then an item detectionsystem may detect items related to “The Office,” such as the “SillyJokes App” (mentioned above with respect to FIG. 4A), based on knowledgethat the URL of the webpage 500 corresponds to “The Office” televisionprogram.

Additionally, or alternatively, items may be related to the webpage 500if they are associated with the owner or host of the webpage 500.Therefore, the item detection system may also analyze information on theowner or host of a webpage 500 to determine which items to present inthe discovery frame 501. Yet another manner to detect related items ofthe webpage 500 may include analyzing the primary content presented inthe main portion 502 of the webpage 500 or in another webpage from thesame website as the webpage 500. For example, referring to FIG. 5A, themain portion 502 of the webpage may include celebrity gossip, and inparticular may include a story involving an actor going surfing with anactress. Accordingly, an item in the discovery frame 501 of the samewebpage 500 may be related to surfing, such as the “Surfing App.” Stillreferring to FIG. 5A, the main portion 502 may include another storyabout an actress going to dinner at a new restaurant. As a result, thediscovery frame 501 may also present an item related to findingrestaurants, such as the “Restaurant Finder App.”

In some examples, the selection and sequencing of items presented in thediscovery frame 501 may be dynamically modified as the user moves acursor 503 on the webpage. For example, if the cursor 503 is over ornear the story about the actress going to dinner at the new restaurant,then the discovery frame 501 might only present items related to the newrestaurant or restaurants in general. Alternatively, the location of thecursor 503 may influence the order in which items appear in thediscovery frame 501. For example, referring to FIG. 5A, because thecursor 503 is located closer to the story about the actress going to thenew restaurant than to the surfing story, the “Restaurant Finder App”may be positioned above the “Surfing App.” To avoid excessive reorderingof the items within the discovery frame 501, the cursor 503 may have tobe within a certain proximity to a particular part of the primarycontent (e.g., the surfing story) for a certain period of time beforethe items are reordered. Further, in some cases, certain items may belocked into their position (e.g., a specific item may be guaranteed atop position).

In order to associate a portion of a webpage 500 with related items, thewebpage 500 may contain elements (e.g., HTML elements) that havespecific context information or explicit item associations. As a result,when a cursor is moved within some proximity to the element or theelement is clicked, the element may invoke processing to determinerelated items for the discovery frame 501. In some embodiments, thewebpage 500 itself (without additional elements for context informationor explicit item associations) can determine changes in context based oncursor location (e.g., mouseover events) or click events, and then callthe second screen experience manager 340 to determine the appropriatelist of items for the discovery frame 501. Also, the webpage 500 maycollect (e.g., at load time or prior thereto) all items based on thecontext information of the webpage 500 so that relatively rapidadjustment of the order of the items in the discovery frame 501 may beachieved.

FIG. 5B shows a computing device 200 (e.g., a tablet) configured topresent an electronic publication 510, such as an electronic book,electronic newspaper, electronic magazine, etc. Alongside of, or overthe top of, the electronic publication 510, the computing device 200 maydisplay a discovery frame 511. The discovery frame 511 may include itemsrelated to the content of the electronic publication 510. For example,in FIG. 5B, the electronic publication 510 may include a “Harry Potter”book in digital form, and therefore, items associated with the “HarryPotter” book may be presented in the discovery frame 511. The “HarryPotter” book may be associated with an application for teaching magictricks called a “Magic Trick App,” in a case where the book is aboutchildren who have special powers. That is, the items in the discoveryframe 511 may be generally related to the electronic publication 510.However, more specific relationships between the content of theelectronic publication 510 and the items of the discovery frame 511 mayexist. For example, the related items may include a “Fishing App” item,if the content on the page being shown (e.g., page 39) describes acharacter going fishing. Context information may be embedded in variousparts of the page so that the related items for that page may bedetected. In such embodiments, where the content of the page (or portionbeing displayed) impacts which items are presented, it should berealized that as a user navigates from one page to another (or scrollsfrom one portion of a page to another portion), the items presented inthe discovery frame 511 may change since the context information of thecontent being consumed may change. Also, context information may beextracted and used to determine related items when a user highlights aword or phrase. In other words, highlighting, or otherwise selecting, aword or phrase in the electronic publication 510 may trigger thecomputing device 200 to use the highlighted text as context informationto retrieve a list of items explicitly or implicitly associated with thehighlighted text.

FIG. 5C illustrates a webpage 520 that may be displayed on a firstscreen device 301, a second screen device 302, or other computing device200 using a web browser. In some examples, the webpage 520 may belong toa website hosted by the second screen experience manager 340.

Referring to FIG. 5C, the webpage 520 may include one or more user inputfields 521 configured to receive user input (e.g., keywords) entered bya user through, for example, a keyboard, microphone, etc. The webpage520 may also include a search button 522 configured to, when selected,cause an item detection system (described in more detail below) tosearch for related items (e.g., applications/computer programs) based onthe user input in the user input field(s) 501.

After the search button 502 is selected, a new webpage 530 may bedisplayed. The new webpage 530 may display detected items based on thesearch results. For example, referring to FIG. 5C, if a user enters “TheOffice” into a field 521 for capturing a content name (e.g., televisionprogram name) and selects the search button 502, then the webpage 530may present one or more items related to “The Office,” which is atelevision program provided by NBC. In some examples, the displayeditems may be displayed in an order based on a rating or degree ofrelevance. The rating or degree of relevance may take into considerationuser preferences of the user operating the web browser that renders thewebpage 530.

FIG. 6 is a diagram illustrating an example configuration of an itemdetection system 600 that may detect relevant items based on contextinformation of primary content, such as, for example, a title of aselected piece of content in a program guide 400 as described withrespect to FIG. 4A or the user input entered in a webpage 500 asdescribed with respect to FIG. 5C. The item detection system 600 may beembodied within the local office 103 of FIG. 3, the second screenexperience manager 340 of FIG. 3, or one or more other computing devices200 connected elsewhere on the network 330 of FIG. 3. As shown in FIG.6, the item detection system 600 may include an item database 601, aquery engine 602, a targeting engine 603, a mapping engine 604, and asettlement engine 605.

The item database 601 may store item data, including one or more items(e.g., applications, audio/video content, links, etc.) and iteminformation (e.g., context information, an item ID, author information,platform information, rating information, etc.) in association with eachitem. Item information may be used to determine which items are ofinterest. For example, item information may be searched to identify oneor more items, and those items may be displayed in a discovery frame 405of the program guide 400. Table 1 below shows a graphical representationof example item data stored in the item database 601.

TABLE 1 Item Data Item/ Item Item Information Name ID Content ChannelKeywords Platform Rating The Office 1 The Office NBC Comedy Xbox 2.5Video TBS Steve Game Carell Video game Trivia NBC App 2 30 Rock NBC NBCiOS 4.5 Apprentice Community The Office . . . NBC App 3 30 Rock NBC NBCAndroid 4.3 Apprentice Community The Office . . . NBC App 4 30 Rock NBCNBC Windows 4.5 Apprentice Phone Community The Office . . . The Office 5The Office NBC Comedy Any web 3 Shop TBS Steve browser Carell Watch 6 30Rock NBC Tablet iOS 5 With Me Apprentice Local iPAD Community TBS ExtraThe Office CBS Twitter Local Prog. FOX Facebook Conan TNT CommentsBaseball USA . . . . . . Silly Jokes 7 30 Rock NBC Comedy iOS 3.5 AppCommunity TBS The Office Spike Bonus 8 The Office NBC MPEG2 3.7 Clip!

Table 1 includes items corresponding to the items shown in FIG. 4A. Asshown in Table 1, each item may be associated with item information.Table 1 illustrates a number of columns for various types of iteminformation. In the “ID” column, item information may include an itemidentification number that may be unique to each item. This number maybe generated according to an algorithm so that identification numbersare sequentially assigned or may be generated randomly. The itemidentification number may be used to reference items so that items canbe transferred between devices and/or modules. In the “Content” column,item information may include a name of primary content (e.g., atelevision show/program, movie, etc.) associated with the item. Forexample, the “Watch With Me” item may be associated with one or moreshows that supplemental content is available for. In other words, if the“Watch With Me” item includes supplemental content for the televisionshow “30 Rock,” then the item information may include “30 Rock” (or someother type of data identifying the “30 Rock” program) in the “Content”column associated with the “Watch With Me” item. However, if the “WatchWith Me” item does not include supplemental content for the televisionshow “30” Rock,” the “Content” column associated with the “Watch WithMe” item might not identify “30 Rock.” In the “Channel” column, iteminformation may identify a logical channel (e.g., NBC, channel 4, etc.)associated with a particular item. For example, the “NBC App” item maybe associated with a logical channel carrying content provided by theNational Broadcasting Company (NBC). As a result, whenever a userscrolls over a program shown on a logical channel carrying contentprovided by NBC, the “NBC App” may appear in the discovery frame 405 ofthe program guide 400. In the “Keywords” column, item information mayinclude a plurality of words, terms, phrases, etc. that may be used todescribe or otherwise identify an item. Users may set preferences thatmay be compared to item information in the “Keywords” column to helpusers identify items that are related to their preferences. The iteminformation included in the “Keywords” column may be set by a providerof the item or another administrator, or administrative organization, sothat the associated item is more likely to be discovered by a targetedgroup of consumers. For example, the creator of an application forsearching for travel deals may include the word “travel” as iteminformation in the “Keyword” column for that application, so that userswho specify that they enjoy traveling are more likely to discover theapplication than users who do not specify that they enjoy traveling (orspecify that they do not enjoy traveling). In the “Platform” column,item information may indicate which platforms the item is compatiblewith. As shown in Table 1, three different versions of the “NBC App” mayexist for three different platforms: Apple's iOS™, Google's ANDROID™,and Microsoft's WINDOWS PHONE™. In some cases, a single item may becompatible with a plurality of platforms. Here, the term “platform” isnot limited to operating systems, but may also specify file extensionsor standards. Further, it is contemplated that various devices may besuited for various platforms, so the item information in the “Platform”column, or another column, may indicate which devices are compatible orsuited for the associated item. In the “Rating” column, item informationmay include a rating associated with the item. This rating may be basedon consumer feedback. In some cases, a single item may be rated for avariety of characteristics. For example, an application may have asports rating and a separate music rating. Therefore, the iteminformation may be used to detect items for users who like sports andusers who like music. As a result, a user who identifies himself as moreof a sports person than a music person may be exposed to an item havinga relatively high sports rating although its music rating is relativelylow.

It should be understood that Table 1 illustrates example relationshipsof various information stored in memory, and that these relationshipscould also be illustrated with more than one table. For example, therecould be three tables where a first table includes metadata inassociation with items, a second table includes identifiers inassociation with content, and a third table includes links between thefirst table and second table. In some examples, storage capacity may beconserved with a three table arrangement as one entry in the first tablecan link to multiple entries in the second table and one entry in thesecond table can link to multiple entries in the first table. Thisarrangement may be particularly desirable where there are similar itemsfor multiple platforms. It should also be understood that Table 1 aboveillustrates only some of the types of item information that may bestored in the database 601. In other embodiments, more or less iteminformation may be stored. For example, an additional column may includean Entertainment Identifier Registry (EIDR) identifier associated witheach piece of content. Also, each item does not have to include eachtype of item information. Some items may have data for each type of iteminformation while some items may have data for just some of the types ofitem information. This is illustrated in Table 1 by the empty cell inthe “Keywords” column for the “Bonus Clip!” item.

The database 601, regardless of its contents, may be configured toconnect to the query engine 602 of the item detection system 600. Thequery engine 602 may be a module of code, including computer-executableinstructions, executed by a processor of a computing device 200. Thequery engine 602 may be configured to receive context information from auser device 610, which may be a first screen device 301, second screendevice 302, interface 120, or any other computing device 200. Thecontext information may include any type of data that can be used tosearch the database 101. In other words, context information may includeany type of data that associates an item to content. For example, thecontext information may be an identifier that identifies a selectedcontent or portion thereof. Specifically, the context information mayinclude the name of a television program selected using a program guide400. Another example of context information may be the name (e.g., NBC)or number (e.g., channel 4) associated with a logical channel number orservice identifier that a user is scrolling over within the programguide 400. Still another example of context information may includekeywords (e.g., “video games,” “football,” “trivia,” etc.), a URL (e.g.,“http://www.youtube.com”), or the name of a book or other publication(including those available in digital form) entered into user inputfields 501 of a webpage 500. In some examples, context information mayalso include user preferences associated with a user of the user device610 or the user device 610 itself. Additionally, or alternatively, insome cases, the context information may include information about theuser device 610, such as what operating system the user device 610 isrunning, what type of display the user device 610 is coupled to, whattype of video card the user device 610 includes, or any otherinformation pertaining to the software and/or hardware of the userdevice 610. The following provides a non-exhaustive list of examples ofcontext information: a content title (e.g., television program title,electronic book title, etc.); content description; content format (e.g.,image, video, etc.); content file-type (e.g., .avi, .divx, .mov, etc.);genre (sports, comedy, romance, reality television, etc.); contentprovider; keywords or search string; location in program guide 400 orother screens; location in webpage 500; information regarding otheritems; program ID; episode ID; series ID; actor/actress names;producer/director names; paid associations (sponsorships); item ID(where item owner/creator explicitly designates an item as correspondingto content); program closed captioning feed; video data of content;audio data of content; similar asset consumption data (those who likedthis content consumed this application); time of day; demographics ofuser; etc.

Using the context information, the query engine 602 may search thedatabase 601. The query engine 602 may prompt a user for additionalinformation if no search results are found. If search results are found,the query engine 602 may return the results to the user device 610.Also, if there are search results, the search results may include one ormore items that were identified as matching the context information. Forexample, a plurality of items may be returned in response to a searchusing the contextual information even where the contextual informationonly includes a single piece of information, such as a televisionprogram name. In some embodiments, the number of items of the searchresults may vary depending on the user device 610 to which the searchresults are transmitted. For example, referring to FIG. 4A, if theprogram guide 400 is configured to display only four items, then thequery engine 602 may only send four items (which may be selected basedon a rating). To accomplish this, the initial request for related items(e.g., the request including the context information) may also indicatehow many results are desired (e.g., how many slots are available fordisplaying related items). Alternatively, the query engine 602 mayidentify the requesting device, type of requesting device (e.g., whetherit is a smartphone, tablet computer, personal computer, etc.), or thetype of frame for presenting the items (e.g., whether it is discoveryframe 405 of FIG. 4A, detailed information frame 412 of FIG. 4B,discovery frame 422 of FIG. 4C, etc.), and may consult a database todetermine how many items can be displayed for that requesting device,type of requesting device, or type of frame. In other embodiments, thequery engine 602 may be configured to transmit up to a certain number ofitems (which may vary based upon heuristics), and the user device 610may be configured to select which items to present. Further, the queryengine 602 may perform more than one search at the same time. Moreover,the query engine 602 may be configured to perform one or more of avariety of search algorithms.

As shown in FIG. 6, the item detection system 600 may include atargeting engine 603 as well. The targeting engine 603 may be a moduleof code, including computer-executable instructions, executed by aprocessor of a computing device 200. The targeting engine 603 may beconfigured to assist the query engine in performing the search so thatthe search results are targeted to the user submitting the contextinformation via the user device 610. Specifically, the targeting engine603 may filter search results or modify search parameters so that,search results based on context information may vary according todemographics, geographic location, time of day, and other suchinformation. In other words, the targeting engine 603 may leveragedemographic information, which may be generated based on surveyinformation, to filter search results to remove items that are notcommonly desired by a certain population, when a member of thatpopulation is to receive the items. For example, if context informationincludes information indicating that a user is a child, the targetingengine 603 may filter the results of the query engine 602, so that theresults returned to the user may include items targeted for children.

Further, the mapping engine 604 may be configured to provide aninterface for entities that wish to register items with the database601. The mapping engine 604 may be a module of code, includingcomputer-executable instructions, executed by a processor of a computingdevice 200. As shown in FIG. 6, the mapping engine 604 may be configuredto communicate with a network 630. The mapping engine 604 may provide aweb portal which entities may access using their own devices via thenetwork 630. Once at the web portal, an entity may submit an item (e.g.,an application) to be uploaded to the database 101. At that time, theentity may also submit other item information to be associated with theitem. For example, the entity may submit a program ID of a program thatthe entity wishes its item to be associated with. As a result, an entityproviding an item may control which pieces of content their items areassociated with. In other cases, an entity may select a genre (e.g.,science fiction, sports, reality television, etc.) that it wishes toassociate its item with.

The mapping engine 604 may determine whether an item is accepted forregistration with the database 601. In some examples, an item may beaccepted if it meets certain criteria, such as it is an acceptable typeof file, does not exceed a certain file size, and/or terms andconditions have been agreed to by the creator of the item. When themapping engine 604 determines that an item is accepted, the mappingengine 604 may cause the item and its associated item information to bestored in the database 601. The mapping engine 604 may also allow theitem and associated item information to be subsequently edited.Additionally, the mapping engine 604 may assign priorities and/orratings to the items entered into the database 601. The priorities andratings of the items may be based on a subscription tier of the entityproviding the item. For example, if the provider of an item is apremium-level customer, the item may be given a higher priority and/orrating so that the item is more likely to be included in search resultsobtained by the query engine 602. The level/tier of an entity may bebased on popularity, trustworthiness, etc. of the entity and/or feescollected from the entity or from end users of the items. Therefore, themapping engine 604 may be configured to determine the level/tier ofvarious entities based on these characteristics.

To obtain information for making determinations based on suchcharacteristics, the mapping engine 604 may use a settlement engine 605.The settlement engine 605 may be a module of code, includingcomputer-executable instructions, executed by a processor of a computingdevice 200. The settlement engine 605 may track which items are showingup in search results of the query engine 602 and/or which items arebeing transferred from the query engine 602 to end users. The settlementengine 605 may also be configured to track placement of items and feescollected from the entity and/or end users of the items. For example,the settlement engine 605 may determine that an entity, whichcreated/supplied a particular item, owes a certain amount of money basedon placement of the particular item in a program guide 400. Thesettlement engine 605 may determine the identity of the entity with theassistance of the mapping engine 604, which may determine which entitiesare associated with which items.

FIG. 7 is a flow diagram illustrating an example method of the presentdisclosure. The flow diagram illustrates steps that may be performed,for example, by the item detection system 600 and/or computing devices200 which may be in communication with the item detection system, suchas first screen devices 301 and second screen devices 302.

In step 701, context information may be obtained to determine ordescribe criteria that will be used in a search to find related itemsthat may be of interest to the user. The manner in which the contextinformation is obtained may vary according to the particular embodiment.For example, referring to FIG. 4A, context information may be obtainedby extracting context information (not shown) embedded within thecontent options 401. Specifically, when a user navigates to a contentoption 401, a script (e.g., computer code, such as a JavaScript) withinthe program guide 400 or another application interface, or a computingdevice 200, such as an interface 120 or the first screen device 301, mayidentify the selected content option 401 and extract the contextinformation for that content option 401. Also, referring to FIGS. 5A and5B, various context information may be embedded throughout other sourcessuch as the webpage 500 and electronic publication 510, respectively.Context information may be extracted in some examples of step 701 aswell, so that items related to particular portions of the webpage 500and electronic publication 510 being consumed may be detected. Asdescribed above, the context information may include any type of data,such as a content name (e.g., television program name), contentidentifier, genre, event identifier (e.g., fishing), etc. that can beused to detect items. Accordingly, in the example of FIG. 4A, step 701may be performed to obtain a content title—“The Office”—by extractingsuch title from the selected content option 401.

Once context information is obtained, it may be transmitted to andreceived by the item detection system 600 at step 702. For example, thecontext information may be received by the query engine 602 of the itemdetection system. The item detection system 600 may receive the contextinformation via various connections, including the links 101 and network330. For example, where the item detection system 600 is located at thelocal office 103, the context information may be transmitted in theupstream bandwidth of the links 101 from the interface 120 through theTS 104 to the item detection system 600. In another example, where theitem detection system 600 is located within or associated with thesecond screen experience manager 340, the context information may betransmitted wirelessly from a second screen device 302 to the secondscreen experience manager 340, for example, through the network 330.

In step 703, the received context information is used to detect itemsrelated to the content from which the context information is obtained.This detection may comprise searching a database 601 using the contextinformation. When detecting related items, the item detection system 600may be configured to search the database 601 until a certain number ofrelated items are detected or to search the entire database 601 andchoose a certain number of items from among all the items identified ashaving some relevance, e.g., a predetermined level of relevance. In someexamples, an item might only be detected as a related item if it meetstwo or more criteria. For example, an item might only be detected oridentified if it is operable on a particular platform and has a keywordmatching one of the corresponding criteria in the context information.Various embodiments may use different algorithms for performing thesearch and detecting items at step 703.

In step 704, the item detection system 600 may rank or arrange items inorder based on various factors. One such factor may be a rating, whichmay be determined based on user feedback. For example, users may rate aparticular item as deserving 4 out of 5 stars, and thus, the particularitem may have a 4.0 rating. Another factor may be a priority associatedwith the item, which may be determined based on a status (e.g., premiumstatus) of an entity that created or provided the item. For example,some entities may subscribe for premium status so that the items theysubmit have a higher priority, and therefore, may have a higherlikelihood of being distributed. Yet another factor may be a degree ofrelevance, which may be a score representing how similar an item is tothe received context information. Demographic and geographicalinformation associated with the context information as well as time ofday information of when the context information is provided to the itemdetection system 600 may also be factored into the arrangement. The itemdetection system 600 may determine the demographic and geographicalinformation based on the user device 610 providing the contextinformation and/or based on consumption data of a user using the userdevice 610. Still another factor may be user preferences, which may beprovided by a user. Weights may be assigned to one or more of thefactors so that a final order may be determined.

In step 705, the detected items, which may be arranged in step 704, maybe delivered to the user device 610 that supplied the contextinformation used to detect the items. For example, the detected itemsmay be returned to an interface 120 that supplied the contextinformation. Alternatively, the detected items may be delivered toanother computing device 200 associated with the user device 610 thatsupplied the context information. For example, a second screen device302 may extract context information from an interface such as a programguide 400 and send the context information to the item detection system600, which may deliver detected items to a first screen device 301associated with the second screen device 302.

The user device 610 that receives the detected items may then presentthe items at step 706. The items may be presented within a designatedportion of a screen of the user device 610. For example, referring toFIG. 4A, a first screen device 301 or second screen device 302 mayreceive the items 406 and present them in the discovery frame 405.Similarly, in the example embodiment of FIG. 4B, a first screen device301 or second screen device 302 may receive items and present them inthe detailed information frame 412. In the example embodiment of FIG.4G, the interface 120 or first screen device 301 may receive thedetected items and present them in the discovery frame 480 overlayingthe primary content playing on the first screen device 301. Also, in theexample embodiment of FIG. 5A, a web browser of a computing device 200may receive the detected items and present them in the discovery frame501 of the webpage 500. Or, in the example embodiment of FIG. 5B, thecomputing device 200 (e.g., an electronic book reader) may receive thedetected items and present them in the discovery frame 511.

The item detection system 600 discussed above may assist users indiscovering a number of applications, such as a supplemental contentpresentation application (also referred to herein as the “Watch With Me”application), that are related to a primary piece of content that theuser is consuming. Below, further details describing the “Watch With Me”application are disclosed. The “Watch With Me” application may includecomputer-executable instructions that, when executed by at least oneprocessor 201 of a computing device 200 (e.g., a second screen device302), may cause the computing device 200 to render a second screen forpresenting supplemental content related to primary content that is shownon a first screen device 301. The “Watch With Me” application may belaunched/downloaded by any of the methods disclosed herein, such as byselecting the application from a program guide 400. The “Watch With Me”application may also be launched from within another application, andthus, the “Watch With Me” application could refer to one or more modulesof code or scripts (e.g., JavaScripts) within a larger application. Forexample, the “Watch With Me” application could be launched when a useruses another application on the second screen device 302 to tune thefirst screen device 301 to a channel. At that time, the otherapplication may determine whether supplemental content for the primarycontent on the tuned to channel is available through the “Watch With Me”application. If so, the other application may launch the “Watch With Me”application so that the supplemental content for the tuned to channel ispresented on the second screen device 302.

FIG. 8 is a diagram illustrating another aspect of the disclosure.Specifically, FIG. 8 illustrates an example second screen 800 presentedon a second screen device 302. In the example embodiment of FIG. 8, thesecond screen 800 presents supplemental content related to primarycontent, such as a live television program called “The Voice,” that isbeing consumed on a first screen device 301, such as a television. Asshown, the supplemental content presented on the second screen 800 mayinclude a quote from an individual on the show (e.g., Cee-Lo) along withan image of the individual. Example forms of supplemental content thatmay be presented include images (e.g., images from the show, picturestaken by people at the show, etc.), audio (e.g., a recording of a songsung by the original singer), video, and/or text.

The second screen 800 may also include a timeline 801. The timeline 801shown on the second screen 800 may correspond to the piece of primarycontent (e.g., a television show, sporting event, etc.) being consumedon the first screen device 301. In the example of FIG. 8, the timeline801 corresponds to an episode of “The Voice.” If the second screendevice 302 were controlled to display another second screen for anotherpiece of primary content (e.g., a different television show or adifferent episode of “The Voice”), the timeline displayed on the othersecond screen might include different data.

The timeline 801 may provide a graphical, linear representation ofevents and the chronological order in which they occur within thecorresponding piece of content. Herein, the timeline 801 may refer tothe graphical representation and/or the data (or computer-executableinstructions) used to render the timeline 801. Further, the timeline 801may demonstrate the relationship between a point in time of the primarycontent and the supplemental content presented on the second screen 800.Referring to FIG. 8, a left end of the timeline 801 may represent abeginning of the primary content (e.g., a time point of zero minutes)while a right end of the timeline 801 may represent an end (or estimatedend) of the primary content (e.g., a time point of 60 minutes). Thetimeline 801 may include a shaded portion 802 that changes in lengthalong the timeline 801, and the rightmost edge of the shaded portion 802demonstrates a point in time of the primary content for whichcorresponding supplemental content is being presented on the secondscreen 800. In some examples, the timeline 801 may include a secondshaded portion (not shown), which may be a different color or shade fromthe shaded portion 802, to demonstrate where along the timeline 801 theprimary content playing on the first screen device 301 is at.

In some examples, the supplemental content presented on the secondscreen 800 may be approximately synchronized with the primary contentbeing presented on a first screen device 301 so that the supplementalcontent may change along with the primary content. In such cases, as theprimary content progresses (e.g., as “The Voice” continues), the shadedportion 802 may extend in length to illustrate the time point of theshow. For example, referring to FIG. 8, if the timeline 801 weresynchronized with the playback of the primary content, the timeline 801may indicate that the primary content has been playing for 24 minutesand 51 seconds. Although this time is shown in FIG. 8, this time mightbe approximated by the user based on the length of the shaded portion802 and the time might not be shown or might only be shown upon request.

Notably, a user may interact with the timeline 801 to select differentpoints in time along the timeline 801 thereby causing different portionsof the supplemental content to be presented on the second screen 800. Inother words, the timeline 801 does not have to stay synchronized withthe primary content. Referring to FIG. 8, a user may move the rightmostedge of the shaded portion 802 to a point along the timeline 801 forwhich the user would like to consume (e.g., view) supplemental content.This may be done by selecting keys on the second screen device 302,selecting virtual keys on the second screen 800, using a finger orstylus where the second screen 800 is a touch-screen, etc. In thismanner, the user can jump around to different points along the timeline801 to see different portions of the supplemental content. However, whenthe user would like to return to a state where the supplemental contentis synchronized with the primary content, the user may choose a resumekey (not shown) that automatically adjusts the timeline 801 to return tothe point in time of the corresponding primary content. Similarly, theuser may also operate pause, play, fast-forward, rewind, and other keys(not shown) to adjust the timeline 801 and cause the presentation of thetimeline's supplemental content to advance, fast-forward, rewind, orpause in a similar fashion as can be done with the primary videocontent. So in some embodiments, a user may have the option ofdynamically viewing supplemental content during playback of the primarycontent, with the supplemental content synchronized to the playback ofthe primary content, while in other embodiments the user may have thepresentation of supplemental content dynamically occurring on a schedulethat is a few minutes ahead of the actual playback of the primarycontent (allowing the user a minute to view and digest the supplementalcontent before the corresponding time point in the primary content isplayed on the first screen).

Additionally, the timeline 801 may include markers 803 configured tomark certain points along the timeline 801. When a marker 803 is active(e.g., when the playback point in the primary content has reached themarker point in the timeline), it may cause a particular piece ofsupplemental content to be presented on the second screen. Referring toFIG. 8, the active marker is shown as a shaded marker 803 s. It is thisshaded marker 803 s which causes the second screen 800 to present thequote shown as supplemental content. In some examples, there might beless supplemental content than primary content, and so the supplementalcontent may remain on the second screen 800 until it is appropriate topresent the next piece of supplemental content. In other words, theremay be varying intervals between the markers 803 of the timeline 801, sosupplemental content corresponding to an active marker from among themarkers 803 may be presented for a period of time until a next marker803 becomes active. For example, in FIG. 8, the supplemental contentcorresponds to the shaded marker 803 s (which is the most recentlyactivated marker) although the shaded portion 802 of the timeline 801shows that the primary content is at a point in time past the shadedmarker 803 s.

In FIG. 8, the markers 803 are illustrated as triangles; however, othershapes, colors, etc. may be used. In some embodiments, there may be twotypes of markers: first markers 803 a and second markers 803 b. Firstmarkers 803 a may include markers created by an operator, a contentcreator, a content service provider, or some other administrativeentity. In contrast, second markers 803 b may include markers created bya user of the second screen device 302 presenting the timeline 801. Inthe example embodiment of FIG. 8, the first markers 803 a are shownabove the timeline 801, while the second markers 803 b are shown belowthe timeline 801. In other embodiments, the first markers 803 a andsecond markers 803 b may appear on the reverse sides of the timeline801. Alternatively, the first markers 803 a and second markers 803 b maybe on the same side of the timeline and different colors or shapes maybe used to distinguish them.

The timeline 801 of FIG. 8 may be presented on a plurality of secondscreen devices 302. For example, in a case where two different users areconsuming similar content (e.g., the same episode of “The Voice”) on afirst screen device 301 and both users have the “Watch With Me”application running on their respective second screen devices 302, thesecond screen devices 302 may each present the timeline 801. Eachtimeline 801 may include the same first markers 803 a, however, eachtimeline 801 may have different second markers 803 b. As describedfurther below, each timeline 801 may be modified. Modifications madewith respect to the first markers 803 a may impact each timeline 801,while modifications made with respect to the second markers 803 b mightonly impact the respective timeline 801.

The first markers 803 a may be pre-set and/or added in real-time. Wherethe first markers 803 a are pre-set, the timeline 801 may include thefirst markers 803 when it is initially rendered by the second screendevice 302. A provider of the primary content may know that a particularevent will take place in the primary content at a particular point intime, and therefore, may create a marker 803 corresponding to thatparticular point in time that will trigger a specific portion ofsupplemental content to be shown in the second screen 800. For example,a provider of primary content may know that an actor will be driving aparticular car at a certain time during a television program, andtherefore, may create a marker 803 on the timeline 801 that causes awebpage, video, etc. that is related to the car to be presented on thesecond screen 800.

As mentioned above, the first markers 803 a may also be added inreal-time. An administrative entity may be designated to monitor livecontent to identify interesting events. The administrative entity maydetermine that a particular event has occurred within a piece of primarycontent, and may desire to supply additional supplemental content forthat event. If so, the administrative entity may send a signal (or causea signal to be sent) to a second screen device 302 presentingsupplemental content for that primary content so that a first marker 803a may be added to the corresponding timeline 801. The administrativeentity may also maintain a copy of the first markers 803 a in its ownmemory, as well as information for the supplemental content (e.g., theapplication and/or image files for an interactive application that is toappear at the three minutes and ten seconds (3:10) mark in a televisionprogram) and information identifying how/where the supporting files forthe supplemental content may be retrieved, for presentation to users whorequest the same primary content in the future. As a result, when theadded first marker 803 a is selected, the second screen device 302 maybe controlled to present a particular piece of supplemental content.

The second markers 803 b may also be added by a user of the secondscreen device 302. When a user determines that an event in the primarycontent and/or the corresponding supplemental content is interesting,he/she may wish to mark that event. To accomplish this, a user mayclick/press on a part of the timeline 801 or a designated mark key 804,and as a result a second marker 803 b may appear on the timeline 801 ata position corresponding to the current point in time in the primarycontent on the first screen device 301. In this manner, a user may beable to identify supplemental content for later consumption. Users mayfind it desirable to mark supplemental content, for example, when theuser is busy consuming the primary content and does not want to bedistracted for the time being, but wants to take a look at thesupplemental content in more detail later. Also, the second screendevice 302 may be used to replay part, or all, of the primary contentcorresponding to any of the pieces of supplemental content, and the usermay mark the timeline to indicate points in time of the primary contentthat he/she would like to view again at a later time.

In addition, second markers 803 b may also be generated automatically inresponse to an action of the user. For example, a user may react tocertain primary content or supplemental content by selecting an optionto share the comment with one or more friends in a social network, andas a result, a second marker 803 b may be added to the timeline 801 at atime corresponding to when the user shares the comment. For example, auser may select a share option 805 on the second screen 800 in responseto reading the Cee-Lo quote shown in FIG. 8, and this selection maycause the second screen device to render a user input window 901 asshown in FIG. 9.

Referring to FIG. 9, the user input window 901 may pop-up onto thesecond screen 800, and overlay the supplemental content presented on thesecond screen 800. When the user input window 901 is shown, a keypad 902may also be shown on the second screen 800. Using the keypad 902, a usermay enter data (e.g., comments) into the user input window 901 inresponse to the supplemental content and/or an event within the primarycontent. The user may then choose to share the entered data with others.In some examples, the entered data may be shared through a socialnetwork service, such as FACEBOOK™, TWITTER™, TUNERFISH™, etc. Forexample, if the entered data is shared through FACEBOOK™, the entereddata may appear as a wall post on a facebook page of the user. A usermay choose which social network services he/she would like to share theentered data through. In some cases, the user may enter data once, andshare that data through a plurality of social network services.

If the user enters and shares data, a second marker 803 b may be addedto the timeline 801 at a point in time when the user selected the shareoption 805. Notably, a user may enter data at multiple times therebycreating a plurality of second markers 803 b along the timeline 801. Asa result, the user may be able to subsequently review portions of theprimary content and/or supplemental content that he/she entered data for(e.g., portions the user commented on). Thus, the timeline 801 mayinclude a log of data entries.

Further, when a user is finished consuming primary and/or supplementalcontent (whether because the user has viewed all of the content or theuser chooses to stop consuming the content), the first markers 803 aand/or second markers 803 b of the timeline 801 may be stored. Thetimeline 801 may be stored in a storage area on the second screen device302 or in a storage area on the network 330 (e.g., in the second screenexperience manager 340) to which the second screen device 302 isconnected. If a user desires to view the timeline 801 at a later time(e.g., when watching a rerun of the primary content, or when resumingplayback of the primary content via a DVR), the user may view theprevious data he/she entered by selecting the second markers 803 b.Users may also delete second markers 803 b that they have created. Auser may wish to delete a second marker 803 b after he/she has reviewedthe supplemental content thereof and does not want to review it again.

FIG. 8 illustrates an example of the second screen 800 when a play alongoption 806 a is selected. When the play along option 806 a is selected,the presentation of supplemental content may automatically progress insynchronization with the presentation of primary content on the firstscreen device 301, so that if the user presses pause on either the firstscreen (e.g., via remote control) or second screen device 302 (e.g., bytapping the play along option 806 a again to deselect it), then both thepresentation of the primary content on the first screen device 301, andthe presentation of the supplemental content on the second screen, maybe paused. When the play along option 806 a is not selected, then thepresentation of the primary content on the first screen and thesupplemental content on the second screen may be independent of oneanother, and the second screen content does not advance automatically.Instead, the presentation of supplemental content on the second screenmay be in response to user selection of markers 803 a,b from thetimeline. The play along option 806 a, however, illustrates just oneoption of the second screen experience application. That is, othersecond screen options 806 may exist within the same application that maygenerate different second screens.

FIG. 10 illustrates an example second screen 1000 of the “Watch With Me”application that may be presented on a second screen device 302 when aconversations option 806 b is selected. The “Watch With Me” applicationon the second screen device may receive messages through data feeds fromsocial network services, such as FACEBOOK™, TWITTER™, TUNERFISH™, etc.and/or other news sources. The data feeds (or data transmissions) mayinclude data entered by users registered through a social networkservice. For example, a TWITTER™ user may enter a message (e.g., a“tweet”) and that message may be grabbed by the “Watch With Me”application and displayed on the second screen 1000. Notably, the “WatchWith Me” application may grab certain messages depending on the primarycontent selected. For example, where “The Voice” is selected, the “WatchWith Me” application may retrieve messages (e.g., “tweets”) addressed ata particular show. Specifically, when “The Voice” is selected, the“Watch With Me” application may subscribe to a particular data feed forretrieving one or more messages with the “#TheVoice” hashtag. In anotherexample, selecting “The Voice” may trigger the “Watch With Me”application to retrieve content on a particular YouTube™ channel.

As shown in FIG. 10, the “Watch With Me” application may sort themessages received. FIG. 10 shows that the messages may be sorted so thatmessages associated with usernames identified as friends appear on theleft side while messages from the general public (e.g., “Everyone”)appear on the right side. In some cases, the messages from the generalpublic appearing on the right side may be filtered to reduce the numberof messages presented so that the user is not overwhelmed if a largenumber of messages are received on the data feeds. This is just oneexample of the many ways that the messages may be sorted. Userpreferences, geographical location, demographic information, and time ofday are a few examples of the different types of information that may beused to sort the messages.

Further, the messages may be ordered based on a time that they wereentered through their respective social network services or based on atime that they are received by the “Watch With Me” application throughthe data feeds described above. As the messages are placed in order,they may also be assigned to a point on the timeline 801. In otherwords, the messages may be synchronized with the primary content and/ortimeline 801. Thus, by adjusting the shaded portion 802 of the timeline801, a user may cause the application to pan to a portion of the secondscreen 1000 showing the corresponding messages. For example, referringto FIG. 10, if a user shortens or extends the shaded portion 802 in thehorizontal direction along the timeline 801, the application may scrollup or down in the vertical direction through the list of messages topresent the corresponding messages.

Another aspect of the disclosure includes adding messages to thetimeline 801. A user may want to mark one or more messages so that theuser can refer back to the messages at a later time. To make thispossible, the “Watch With Me” application may provide a post messageoption 1001 next to each message thereby allowing the user to select amessage of interest to the user. When the user selects the message, amarker 803 may be added to the timeline 801. For example, referring toFIG. 10, a selection S1 of the post message option 1001 alongside of themessage created by “Friend4” may cause the second marker 803 b 1 to beadded to the timeline 801. Further, the second marker 803 b 1 generatedas a result of the selection S1 may be added at a location on thetimeline 801 corresponding to the point in time (within the presentationof the primary content) when the selected message was created. In someembodiments, by moving the shaded portion 802 to desired points on thetimeline 801 and selecting post message options 1001, a user may controlwhere on the timeline 801 the second marker 803 b 1 is added. As aresult of these features, if a user at a time later in the primarycontent (e.g., perhaps at a commercial break towards the end of “TheVoice”) would like to see the message by “Friend4” again, the user mayselect the first marker 803 b 1. This may cause the second screen 1000to scroll to the point in the list of messages where the message by“Friend4” exists or may cause a pop-up window (not shown) to appear withthe message by “Friend4” inside.

Notably, the user may create a second marker 803 b for any message onthe second screen 1000. For example, referring to FIG. 10, a selectionS2 of the post message option 1001 alongside of the message created by“username1” may cause a second marker 803 b 2 to be added to thetimeline 801. FIG. 10 also shows that the selection S2 may cause a firstmarker 803 a 2 to be created. As mentioned above, the first markers 803a shown on top of the timeline 801 may be added in real-time by anadministrative entity. If a number of people select the same message orcertain people (e.g., a celebrity) select a message, the administrativeentity may determine to add a first marker 803 a to make notice of themessage so that everyone using the timeline 801 may be alerted to themessage. Essentially, the administrative entity can decide that certainmessages should be featured. The determination by the administrativeentity may be a manual determination made by a group of people belongingto the administrative entity or automatically by a computing device 200on behalf of the administrative entity. In the case that thedetermination is made automatically, the computing device 200 may detecthow many times each message is selected, and compare the number of timesto a predetermined threshold. If the number of times a message isselected exceeds the threshold, the computing device 200 may pushinstructions to the second screen device 302 (or store the instructionsat a designated location from which the second screen device 302 maypull data) to cause the “Watch With Me” application to add a firstmarker 803 a corresponding to the message, and the computing device 200may update its own stored copy of the timeline 801 to reflect theaddition. In some cases, instead of automatically pushing theinstructions to the second screen device 302 (or storing theinstructions to a designated location), the computing device 200 mayflag the message that has been selected a number of times and send themessage to one or more individuals of the administrative entity who maymake the final determination of whether to feature the message and pushthe instructions to the second screen device 302.

FIG. 10 illustrates this concept of adding first markers 803 a forcertain messages. As shown in FIG. 10, if a number of people select thepost message option 1001 alongside the message of “username1,” the firstmarker 803 a 2 may be also be created on the timeline 801. Notably, thefirst marker 803 a 2 may appear on the timeline 801 even when the userdoes not make the selection S2, if a number of other users selected themessage of “username1.” Accordingly, the system may draw the user'sattention to messages that are more popular than others or messages fromcertain individuals that the user might not otherwise notice.

Referring to both FIGS. 8 and 10, the quote by Cee-Lo shown on thesecond screen 800 of FIG. 8 may be the result of a first marker 803 aadded to the timeline 801 by an administrative entity in response todetecting a number of selections of the message by “username1” (whichmay be identified as Cee-Lo's username). More specifically, theadministrative entity, or computing device 200 thereof, may havedetermined that username1's message is popular and should be featured.The administrative entity, or computing device 200 thereof, may packagethe message along with an image of Cee-Lo so that the package, whentransmitted to the second screen device 302, may allow the “Watch WithMe” application to present the image and a callout box containing themessage. The “Watch With Me” application may be include a number ofpreset templates, and may be configured to populate those templates withthe data provided by the administrative entity.

In FIG. 10, the timeline 801 shows the same first markers 803 a thatwere shown in the timeline 801 of FIG. 8, despite the fact that thescreens of FIGS. 8 and 10 show different supplemental content (FIG. 8shows a celebrity quote in the when the play along option 806 a isselected, while FIG. 10 shows friend discussions when the conversationsoption 806 b is selected). As such, the same timeline 801 may bepresented by the “Watch With Me” application regardless of which secondscreen option 806 is chosen. However, in other embodiments, the firstmarkers 803 a shown on the timeline 801 may change according to theselected second screen option 806. For example, in another case, thetimeline 801 of FIG. 10 might only show first markers 803 a thatcorrespond to messages since the conversations option 806 b is selected.Similarly, the timeline 801 of FIG. 10 might only show second markers803 b that correspond to messages that the user selected since theconversations option 806 b is selected.

FIG. 11 illustrates an example second screen 1100 in which a user hasselected to go back and watch a particular segment of the primarycontent that has already been presented or aired live. For example, theprimary content may be presenting a block of commercials, and the usermay decide that he/she would like to watch an earlier segment of theprimary content that he/she marked. When a user first creates a marker803, a recording command signal may be sent to a digital video recorder(DVR) or a recording server, which may be a computing device 200 on thenetwork 330 (or in the local office 103), to instruct the DVR orrecording server (whichever is used) to record (or buffer) a portion ofthe primary content being presented on the first screen device 301. Insome examples, the portion to be recorded may have a predeterminedduration.

Referring to FIG. 11, the shaded portion 802 of the timeline 801represents that the first screen device 301 is playing a portion of “TheVoice” that is 46 minutes and 35 seconds (46:35) from the beginning. Atthat time, a block of commercials may be playing on the first screendevice 301, and therefore, the user may wish to view an earlier segmentof the “The Voice” corresponding to one of the second markers 803 b.Accordingly, the user may select a desired second marker changing it toa shaded marker 803 s. In response to this selection, the second screen1100 may present the corresponding supplemental content, which may be are-presentation of an earlier segment of “The Voice.” The desiredportion to be re-presented may have been buffered in the user's own DVR,or it may have been stored remotely at a recording server. In caseswhere the portion to be re-presented is buffered in the DVR or recordingserver and desired to be viewed on the second screen device 302, the DVRor recording server (whichever is set up to record the primary contentand support this feature) may transmit the desired portion to the secondscreen device 302 upon receipt of a request for consuming the desiredportion. This request may include an indication of the first screendevice 301, the second screen device 302 to which the desired portionshould be sent, the user or a user account, the primary content, a timepoint for playback, and/or a duration for playback, so that the secondscreen device 302 may receive the correct portion that is desired to bere-presented.

In some cases, the user may wish to play the desired portion on thefirst screen device 301 (e.g., a television). If so, the user may selecta send-to-tv option 1101 on the second screen 1100. This selection ofthe send-to-tv option 1101 may cause the second screen device 302 tosend a signal, including an indication of the first screen device 301,the user or a user account, the desired content, a time point forplayback, and/or a duration for playback, to the DVR or recording server(whichever is set up to record the content and support thisfunctionality). In turn, the signal may cause the DVR or recordingserver to check whether it has recorded the desired portion, and if so,to jump back to a point in time to play the desired portion.Alternatively, the DVR or recording server that receives this signal mayset up a new stream containing the desired portion of primary content,and send a command to a computing device 200, such as the DVR, interface120, or first screen device 301, to force the computing device to tuneto the new service/channel carrying the stream. As a result, the firstscreen device 301 may re-present the desired portion of the primarycontent. After the desired portion is re-presented, the computing device200 may tune back to the service/channel carrying the primary contentthat was being presented before the signal was received from the secondscreen device. While the first screen device 301 is re-presenting adesired portion of the primary content, the DVR or recording server mayrecord (or buffer) the primary content being missed so that when theuser is tuned back to the service/channel carrying the primary content,the user may consume the content from the point in time where he/sheleft off before tuning away to consume the re-presented portion.

In some embodiments, the user may wish to play other types ofsupplemental content (other than re-presentations of the primarycontent) on the first screen device 301 (e.g., a television). If so, theuser may select the send-to-tv option 1101 on the second screen 1100.The “Watch With Me” application may present the send-to-tv option 1101for certain pieces of supplemental content. When the send-to-tv option1101 is selected, the “Watch With Me” application may cause the secondscreen device 302 to communicate with a first screen device 301 orcomputing device 200 connected thereto (e.g., interface 120). The secondscreen device 302 may then transmit the supplemental content via a wiredor wireless connection (e.g., WiFi) to the first screen device 301 or aninterface connected to the first screen device 301 thereby causing thefirst screen device 301 to play the supplemental content. Once thesupplemental content finishes playing, the first screen device 301 mayreturn to the primary content. In some embodiments, the “Watch With Me”application will cause the second screen 1100 to present the primarycontent (which may include commercials) or a related version of theprimary content (e.g., a logo of an advertiser whose commercial wouldhave been shown on the first screen device 301 as part of the primarycontent if the first screen device 301 were not presenting thesupplemental content), while the first screen device 301 is presentingthe supplemental content.

FIG. 12 illustrates an example configuration of a system 1200 forrealizing some of the features of the “Watch With Me” applicationdescribed above. As shown, the “Watch With Me” application may beexecuted on a second screen device 302 of the system 1200. The system1200 may also include the second screen experience manager 340, one ormore social network services 1201, and one or more news sources 1202.

The one or more social network services 1201 may include FACEBOOK™,TWITTER™, TUNERFISH™, etc. Each social network service 1201 may includeany computing device 200, such as personal computers, laptops, tablets,smartphones, PDAs, servers, etc. Any computing device 200 on which auser is signed-in or logged-in may be considered to belong to the socialnetwork service 1201. For example, a user may sign-in to the socialnetwork service 1201 a using a tablet, and that tablet may be consideredto belong to the social network 1201 a as long as the user is signed-inon that tablet. Indeed, the second screen device 302 in FIG. 12 may alsobe considered to belong to one or more of the social network services1201 depending on whether the user has signed-in using the second screendevice 302.

The one or more news sources 1202 may include any computing device 200configured to supply a data feed, such as a rich site summary (RSS)feed. Accordingly, a news source 1202 may include a server that hosts awebpage or website from which a data feed may be received. For example,a news source 1202 may include a website which presents blog posts andbroadcasts blog posts through an RSS feed.

The second screen experience manager 340 may be configured tocommunicate with one or more of the social network services 1201 andnews sources 1202 via the network 330 (e.g., the Internet).Specifically, the second screen experience manager 340 may be configuredto receive one or more data feeds from each of the one or more socialnetwork services 1201 and news sources 1202. Accordingly, the secondscreen experience manager 340 may include an aggregator, such as an RSSreader, to receive and read the data feeds. One or more uniform resourceidentifiers (URIs) may be provided to the aggregator to configure theaggregator to subscribe to certain data feeds. These URIs may beprovided by an administrative entity or other operator of the secondscreen experience manager 340 such that the data feeds may be subscribedto whether or not users have requested supplemental data from the datafeeds. Alternatively, URIs may be provided in response to a request forsupplemental content. For example, referring to FIG. 10, when a usernavigates to a conversations option 806 b to view messages related to“TheVoice,” the second screen device 302 may send a request for suchmessages to the second screen experience manager 340 causing anaggregator therein to be configured with a particular URI forsubscribing to a data feed supplying messages related to “TheVoice.”

Further, the second screen experience manager 340 may be configured toanalyze the data feeds and organize the data. Table 2 below illustratesan example of the various associations that may be created as a resultof organizing the data.

TABLE 2 Social Popularity Network Score of Subject Primary ServiceTimestamp Message Message Username Matter ID Content Facebook ™ 12/7 @Liked Liked by username1 The Voice The Voice 7:33 pm 2,231 facebook pageTwitter ™ 12/7 @ What a Retweeted username2 #TheOffice The Office 7:34pm great 1,043 show . . . Twitter ™ 12/7 @ I love her Retweetedusername3 #TheVoice The Voice 7:34 pm voice . . . 465 tunerfish ™ 12/7 @Best Approved username4 The Voice The Voice 7:34 pm episode by tunerfishyet . . . 988 page

The data in Table 2 is provided to illustrate various aspects related tohow the second screen experience manager 340 might organize varioustypes of data. Here, organizing may include storing the various types ofdata in association with each other. The second screen experiencemanager 340 may include a database, or be coupled to a database,configured to store the various types of data in association with eachother. Referring to Table 2, the “Social Network Service” column mayinclude information identifying one of the social network services 1201.This information may be obtained based on information in the data feedindicating the source of the data feed. For example, where the data feedincludes IPv4 or IPv6 packets, this information may be determined basedon the source address. The “Timestamp” column may include a timestampindicating a date and time for associated information received by thesecond screen experience manager 340 through the various data feeds. Thetimestamp may represent a time the associated information is received,or a time supplied by the social network service 1201 indicating a timethat the associated information was generated or transmitted. Notably,the second screen experience manager 340 may be configured to receiveinformation from different data feeds in parallel, and thus, differentdata may have the same or approximately the same timestamp. The“Message” column includes information entered by a user (e.g., a “tweet”on TWITTER™) or generated in response to a user action (e.g., selectinga “like” key on FACEBOOK™) through one of the social network services1201, and forwarded to the second screen experience manager 340 throughone of the data feeds. Although shown as text in Table 2, the messagemay include images, audio, video, etc. The “Popularity Score of Message”column may include a score (or other valuation) indicating how popular aparticular message might be. The second screen experience manager 340may analyze the messages to compute this score. Such analysis mayinclude tallying a number of times that a message was retransmitted(e.g., retweeted) or a number of times a message was approved (e.g.,liked). This popularity score may be used to determine whether messagesare made accessible through the “Watch With Me” application at all orincluded in the timeline 801 as featured content, as disclosed herein.The “Username” column may include usernames identifying the users of thedifferent social network services 1201 who originally created themessage information received. Information regarding the username may bereceived in association with the respective message information throughthe data feeds. The “Subject Matter ID” column may include informationidentifying the subject matter that the message is directed to. In acase that the social network service is FACEBOOK™, the subject matter IDinformation may indicate a FACEBOOK™ page that was “liked.” Incomparison, in a case that the social network service is TWITTER™, thesubject matter ID information may indicate a hashtag (e.g., “#TheVoice”hashtag) of the message. The subject matter ID information may also bereceived in association with the respective message and username throughthe data feeds. Lastly, the “Primary Content” column includes dataidentifying related primary content that the second screen experienceserver 340 may generate based on data in one of the other columns. Forexample, based on the subject matter ID indicating a FACEBOOK™ page of“The Voice,” the second screen experience server may determine that themessage (e.g., liked) is related to the television program called “TheVoice.”

In addition to communicating with the social network services 1201 asdescribed above, the second screen experience manager 340 maycommunicate with the second screen device 302 executing the “Watch WithMe” application. When the “Watch With Me” application is launched, itmay cause the second screen device 302 to connect to the second screenexperience manager 340 via the network 330. The second screen device 302may send signals to the second screen experience manager 340 indicatingwhich primary content is being consumed and/or requesting specificsupplemental content. Further, the second screen experience manager 340may transmit data, which it receives from the social network services1201, to the second screen device 302. For example, the second screenexperience manager 340 may determine that the second screen device 302should receive supplemental content related to “The Voice.” Then, thesecond screen experience manager 340 may perform a search on theinformation it received from the data feeds and stored by searching fordata associated with primary content indicating “The Voice,” and maytransmit the search results to the second screen device 302 so that thesecond screen device 302 may present a screen such as the second screen1000 of FIG. 10.

Notably, the second screen experience manager 340 may continue to senddata (e.g., messages) to the second screen device 302 so that the secondscreen device may present the second screen 1000 with the most recentmessages. Meanwhile, the second screen experience manager 340 mayreceive reports from the second screen device 302. Such reports mayindicate selections of desired supplemental content, which the secondscreen experience manager 340 may provide in response. Additionally, oralternatively, such reports may indicate selections made by the user.For example, the “Watch With Me” application may cause the second screendevice 302 to report that a user has selected a particular message(e.g., a user has made selection S1 in FIG. 10). Using this informationalong with similar information in reports received from other secondscreen devices 302, the second screen experience manager 340 maydetermine whether a particular message has been selected by a certainnumber of different users. Therefore, the second screen experiencemanager 340 may, to some degree, monitor user inputs on the secondscreen device 302. The second screen experience manager 340 may forwardthe results of its monitoring to an administrative entity or mayautomatically make decisions to generate new markers 803 for thetimeline 801. In any event, the second screen experience manager 340 mayeventually transmit data including instructions to the second screendevice 302 that, when interpreted by the “Watch With Me” application,may cause the “Watch With Me” application to modify the timeline 801 toinclude a new marker. Moreover, the received data may also include themessage or other information to be presented in a case where the addedmarker is selected.

FIG. 13 is a flow diagram illustrating an example method of modifyingthe timeline and presenting messages from data feeds related to aparticular piece of primary content. FIG. 13 also illustrates howinformation from reports related to the messages may be used to modifythe timeline. With respect to the flow diagram of FIG. 13, the steps arediscussed below, for illustrative purposes only, as being performed byor in association with the second screen experience manager 340.However, in other embodiments, one or more of the steps may be performedby another computing device, e.g., device 200, configured to connect tothe network 330 and communicate with second screen devices 302, socialnetwork services 1201, and news sources 1202.

In step 1300, the second screen experience manager 340 may register asecond screen device 302 and/or a user associated with the second screendevice 302. This registration process may include providing asupplemental content presentation application, such as the “Watch WithMe” application, to a second screen device 302. Further, thisregistration step may include receiving information (e.g., a username,password, device ID, etc.) for setting up a user account. When settingup a user account, a user may specify user preferences that maydetermine which timelines and/or which supplemental content aresubsequently provided to the user. For example, a user may specifyhis/her age, gender, interests, etc. so that he/she receives a timelinewith certain pre-configured markers appropriate for that user. As aresult, for example, one user may receive a first timeline with firstsupplemental content for a particular piece of primary content, whileanother user may receive a second timeline with second supplementalcontent for the same piece of primary content.

In addition, while setting up a user account, a user may also provideinformation related to social network services 1201 of the user forconfiguring the second screen experience manager 340 to access thesocial network services 1201. Since the second screen experience manager340 may access the social network services 1201, a user may submitcomments (or other data) to his/her social network services 1201 throughthe “Watch With Me” application running on a second screen device 302.For example, comments may be sent from a second screen device 302 to thesecond screen experience manager 340, which may then forward thecomments to a server of the social network service 1201 along withinformation stored in the second screen experience manager 340 forauthenticating the user to the particular social network service 1201.Further, setting up a user account may configure the second screenexperience manager 340 to store usernames of friends, family, or otherpersons of interest in association with the user account, so that thesecond screen experience manager 340 may filter messages and sendselected messages to a second screen device 302 associated with the useraccount.

Once a user account is set up, a second screen device 302 may beconfigured to interact with the second screen experience manager 340 sothat the remaining steps of FIG. 13 may be performed. For example, theregistration process of step 1300 may configure the second screenexperience manager 340 to push timelines 801 in step 1305, messages instep 1307, or instructions in step 1312 to registered second screendevices 302.

In step 1301, the second screen experience manager 340 may receive oneor more requests for supplemental content. The requests may be receivedfrom one or more first screen devices 301 or from one or more secondscreen devices 302. Specifically, a second screen device 302 may send asignal requesting a particular piece of supplemental content related toa particular piece of primary content, such as an episode of “TheVoice.” In response to receiving the request, the second screenexperience manager 340 may store an identifier identifying the secondscreen device 302 in association with information identifying therequested supplemental content.

In step 1302, the second screen experience manager 340 may receive oraccess one or more timelines 801. The timelines 801 may be provided bycontent creators (e.g., television show producers), the local office103, or other approved entities. In some embodiments, the second screenexperience manager 340 may generate one or more timelines 801. Forexample, the second screen experience manager 340 may use predefinedcode to set up a timeline 801 for a particular piece of content. Morespecifically, the second screen experience manager 340 may determinewhen a timeline 801 is not available for a particular piece of content,and in response, may create a new instance of a module (which mayinclude one or more classes, such as Java classes) ofcomputer-executable instructions designed to present a timeline 801. Insome examples, the second screen experience manager 340 may use acontent listing (e.g., program guide 400) to determine that a particularpiece of content is new, and therefore, that a timeline is not yetavailable and should be generated.

In addition to creating a new instance of a module for a new timeline801, the second screen experience manager 340 may also automatically addsupplemental content to the timeline 801. In some examples, secondscreen experience manager 340 may perform audio and/or image recognitionprocesses on the content for which the timeline 801 is created and mayadd supplemental content according to the results of those processes.For example, audio recognition processes may be performed on the audiodata of the content to determine that an actor in the content refers toa brand of clothing. As a result, the second screen experience manager340 may add a link to a website for that brand of clothing into thetimeline 801. Specifically, the second screen experience manager 340 maycreate a first marker 803 a at a point along the timeline 801representing a time when the actor referred to the brand of clothing sothat that first marker 803 a, when selected, may cause a link to thewebsite for that brand of clothing to be presented on a second screendevice 302. Similarly, image recognition processes may be used toidentify images of an item within the video data of the content so thatlinks to websites selling that item may be added to the timeline 801.Additionally, or alternatively, supplemental content may beautomatically added to the timeline 801 based on the identity of thecontent for which the timeline 801 is being created. That is, based onthe identity of the content, the second screen experience manager 340may determine that predetermined supplemental content associated withthat content should be added to the timeline 801. For example, where thecontent is an episode of “TheVoice,” the timeline 801 may add one ormore first markers 803 a providing additional information about one ormore characters (e.g., Cee-Lo) known to be featured in the content. Inanother example, YouTube™ content on a particular channel known to beassociated with the identified content may be incorporated into thetimeline 801. In some cases, automatically added first markers 803 a maybe inserted into the timeline 801 at points corresponding to knowncommercial breaks within the content so as not to disrupt a user'sconsumption of the content.

In step 1303, the second screen experience manager 340 may receive oneor more data transmissions from social network services 1201 and/or newssources 1202 as described above. As shown in FIG. 13, steps 1301, 1302,and 1303 may be performed in parallel. Thus, the second screenexperience manager 340 may receive a data feed at the same time that itreceives a timeline 801 or request for supplemental content. While steps1301, 1302, and 1303 may be performed in parallel and at the same time,this is not a requirement and the second screen experience manager 340may perform each step whenever it receives the respective data.Moreover, it should be understood that steps 1301, 1302, and 1303 may beperformed continuously or at periodic or variable intervals.

In step 1304, the data from the data transmissions may be analyzed andorganized as described above. Like step 1303, step 1304 may be performedcontinuously or at intervals to organize the data from the data feeds asit is received.

In step 1305, an appropriate timeline 801 (either received, accessed, orgenerated) may be transmitted/delivered to the second screen device 302that sent the request received in step 1301. The appropriate timeline801 may be determined based on information within the requestidentifying the content for which supplemental content is desired oridentifying the supplemental content itself. Here, delivering a timeline801 may include sending data for the timeline 801 and indicating thatthe timeline data is for a timeline 801 so that a supplemental contentpresentation application (e.g., the “Watch With Me” application) runningon a second screen device 302 may configure the timeline 801. In someembodiments, step 1305 may include multicasting one or more timelines801 to all second screen devices 302 in communication with the secondscreen experience manager 340 using one or more multicast signals. Insuch embodiments, each of the second screen devices 302 may determinewhether to buffer and/or present the timelines 801. Where timelines 801are broadcasted, step 1307 might not be performed in response to step1306, and instead, step 1307 may transmit messages continuously orintermittently.

Delivering an appropriate timeline 801 at step 1305 may includedelivering a version of the timeline based on a type of receiving deviceand/or user preferences. There may be multiple timelines 801 for thesame piece of content that are designed for different types of devices.For example, a smartphone may receive one version of the timeline 801while another type of device (e.g., a tablet) may receive anotherversion of the timeline 801. The version of the timeline 801 deliveredto the smartphone might have a smaller layout than the version of thetimeline 801 delivered to the other type of device. Alternatively, theversion delivered to the smartphone may have a vertical layout asopposed to a horizontal layout that is used in the version sent to theother type of device.

In step 1306, the second screen experience manager 340 may receive arequest for messages related to a particular piece of content. Forexample, referring to FIG. 10, a user may select a conversations option806 b. This may selection may trigger the second screen device 302 tosend a request for messages related to a particular piece of content. Inthe case of FIG. 10, where the “Watch With Me” application is set topresent supplemental content for “The Voice,” the request for messagesmay request messages related to “The Voice.” Therefore, the request mayinclude information identifying the content for which messages arerequested.

In response to receiving the request at step 1306, the second screenexperience manager 340 may deliver the messages at step 1307.Specifically, the second screen experience manager 340 may search adatabase or other storage area for messages related to the contentidentified in the request. In cases where the data from the data feedsare analyzed and organized in step 1304, the messages may be more easilyfound by searching for the messages using information identifying thecontent. In some embodiments, step 1307 may include broadcastingmessages to all second screen devices 302 in communication with thesecond screen experience manager 340 using one or more multicastsignals. In such embodiments, each of the second screen devices 302 maydetermine whether to buffer and/or present the messages. Where messagesare broadcasted, step 1307 might not be performed in response to step1306, and instead, step 1307 may transmit messages continuously orintermittently.

Further, in some embodiments, step 1303 might not be performed until therequest for messages is received at step 1306. That is, the secondscreen experience manager 340 might wait until it receives a request formessages related to a particular piece of content, and then maysubscribe to a data feed that provides messages for that particularpiece of content. In such embodiments, the delivering of messages instep 1307 may include subscribing to a particular data feed based on therequest, receiving messages from that data feed, and delivering thosemessages.

In step 1308, the second screen experience manager 340 may receivereports from one or more second screen devices 302. As described abovewith respect to FIG. 12, reports may be sent to indicate that a user hasselected a particular message. The report may identify the particularmessage selected and the user who selected the message, as well as,other information about the user (e.g., geographical location of user,age of user, etc.). Then, at step 1309, the second screen experiencemanager 340 may forward the report to an administrative entity which maydetermine whether the message identified in the report warrants theaddition of a first marker 803 a to the timeline 801 so that the messagemay be featured as described above with reference to FIG. 12.Additionally, or alternatively, the reports may be analyzed by thesecond screen experience manager 340, which may automatically determineif a threshold number of reports identifying the same selected messageare received.

In step 1310, the second screen experience manager 340 may acquireupdates to the timeline 801. That is, the second screen experiencemanager 340 may acquire a command indicating that one or more firstmarkers 803 a should be added to the timeline 801 so that acorresponding piece of supplemental content is shown when each of thefirst markers 803 a is selected. These update commands may include thesupplemental content to be added, a timestamp indicating a location inthe timeline 801 where a marker for the supplemental content should beadded, and information identifying the corresponding content so that thecorrect timeline 801 is updated. Further, these update commands may bereceived from an administrative entity or acquired from the secondscreen experience manager 340 itself when the second screen experiencemanager 340 automatically analyzes the reports in step 1308. Forexample, an administrative entity monitoring a particular piece ofcontent in real-time may detect that an event occurred within thecontent and may decide to provide supplemental content in response tothat event. If so, the administrative entity may provide thesupplemental content along with a command to the second screenexperience manager 340 so that the second screen experience manager 340may modify the timeline 801 accordingly. In another example, theadministrative entity (or the second screen experience manager 340itself) may determine that a particular message should be featured, andmay provide supplemental content, including the message, along with acommand to the second screen experience manager 340 so that the secondscreen experience manager 340 may modify the timeline 801 accordingly.While FIG. 13 illustrates that step 1310 may be performed in response toresults of the analysis at step 1309, FIG. 13 also includes a dottedline arrow to illustrate that updates may be acquired at any time froman administrative entity, which may monitor content in real-time andchoose to create updates to modify timelines 801.

In step 1311, the second screen experience manager 340 may generateinstructions that cause a second screen device 302 to modify a timeline801. For example, when a command is received from an administrativeentity in step 1310, the second screen experience manager 340 maygenerate instructions that cause the second screen device 302 to add afirst marker and corresponding supplemental content to the timeline 801.These generated instructions may include computer-executableinstructions that the second screen device 302 may process or mayprovide information that directs the second screen device 302 to executecomputer-executable instructions therein. In the latter case, theinformation may include the supplemental content to be added, atimestamp indicating a location in the timeline 801 where a marker 803for the supplemental content should be added, and/or informationidentifying the corresponding content so that the correct timeline 801is updated. In the former case, the computer-executable instructions mayinclude this information as well as a script or other module of codethat, when executed, may instruct the “Watch With Me” application on howto modify its timeline 801 to present the additional supplementalcontent.

In step 1312, the generated instructions may be transmitted to a secondscreen device 302. In some cases, the generated instructions may bepushed to the second screen device 302 once they are generated. Thus,second screen devices 302 might not need to request updates in order forthe timeline 801 to stay up to date with the latest supplementalcontent. Alternatively, once the instructions are generated, they may bestored for transmission upon a subsequent request for updates. Forexample, the instructions may be stored in a memory of the second screenexperience manager 340 until the second screen experience manager 340receives a request from a second screen device 302 for updates to itstimeline 801, at which point the second screen experience manager 340may transmit the appropriate instructions.

Although not shown, it should be understood that one or more of thesteps in FIG. 13 may be performed numerous times. For example, afterstep 1312 the process may return to step 1301 to receive another requestfor supplemental content. The subsequent request may be for similarsupplemental content as previously requested (e.g., supplemental contentrelated to the same primary content as the previously requestedsupplemental content) or for new supplemental content related todifferent primary content than the previously requested supplementalcontent. Also, in various circumstances, some steps of FIG. 13 may beperformed more times than others. For example, the second screenexperience manager 340 may receive more reports than requests formessages, and therefore, may perform step 1308 more often than step1306.

Although example embodiments are described above, the various featuresand steps may be combined, divided, omitted, and/or augmented in anydesired manner, depending on the specific secure process desired. Forexample, the process of FIG. 7 may be modified so that step 704 is notperformed, and thus, the items are not arranged in an order. Or, forexample, the process of FIG. 13 may be modified such that the datareceived through the data feeds at step 1303 is not received until aftera request for messages is received in step 1306. This patent should notbe limited to the example embodiments described, but rather should haveits scope determined by the claims that follow.

We claim:
 1. A method, comprising: receiving context informationcomprising information related to a particular piece of content;detecting second screen applications related to the particular piece ofcontent based on the context information; and delivering a link forpresentation on a user device so that the link, when selected, causes adownload or launch of a corresponding one of the detected second screenapplications to a target device.
 2. The method of claim 1, wherein thecontext information is extracted from data embedded in a portion of ascreen of the user device.
 3. The method of claim 1, further comprising:sending a listing of a plurality of pieces of content, the listingindicating at least two different pieces of content having differentcontext information that when received and used to detect second screenapplications yields different second screen applications, wherein thereceiving of context information includes receiving correspondingcontext information for a selected one of the plurality of pieces ofcontent.
 4. The method of claim 1, further comprising: sending a programguide for presentation by the user device, the program guide having aportion with the context information embedded therein; wherein the userdevice is separate from the target device and the target device isconfigured to execute the downloaded second screen application.
 5. Themethod of claim 1, further comprising: providing the particular piece ofcontent, the particular piece of content including a plurality of eventsoccurring in sequence, wherein receiving the context informationincludes receiving information indicating one of the events in responseto that event occurring within the particular piece of content.
 6. Themethod of claim 1, further comprising: receiving metadata for each ofthe second screen applications; storing the metadata in association withthe second screen applications and storing the second screenapplications in association with one or more pieces of content;arranging the detected second screen applications in order based on oneor more factors; and delivering a link for each of the detected secondscreen applications so that the detected second screen applications arepresented in a frame of the screen in accordance with the order, whereinthe detected second screen applications are each configured to presentsupplemental content on second screen devices contemporaneously withpresentation of the particular piece of content on the user device. 7.The method of claim 1, further comprising: delivering item descriptioninformation corresponding to the link, the item description informationincluding a preview of features of the corresponding second screenapplication.
 8. The method of claim 1, wherein the link is presented ina frame of the screen and the frame overlays the content while thecontent is playing.
 9. A method, comprising: delivering a timeline to auser device, the timeline corresponding to a duration of a piece ofcontent and comprising instructions that allow the user device to modifythe timeline to present first supplemental content at a point along thetimeline in response to a user input; acquiring updates to the timeline;and generating, by a computing device, instructions, comprising secondsupplemental content, that cause the user device to modify the timelineto present the second supplemental content at another point along thetimeline.
 10. The method of claim 9, further comprising: receiving adata feed including a plurality of messages; delivering one or more ofthe messages to the user device; and receiving a report from the userdevice indicating a selection of one or more of the messages.
 11. Themethod of claim 10, wherein the receiving of the data feed includessubscribing to a hashtag of a social network service.
 12. The method ofclaim 10, wherein the receiving of the data feed includes subscribing toa rich site summary (RSS) feed of a news source.
 13. The method of claim9, further comprising: receiving a plurality of reports from a pluralityof user devices indicating a selection of one or more messages; andanalyzing the plurality of reports to determine whether a particularmessage from among the one or more messages has been selected a numberof times that exceeds a threshold.
 14. The method of claim 13, whereinthe updates are acquired in response to determining that a particularmessage has been selected a number of times that exceeds the threshold,and the supplemental content included within the instructions includesthe particular message.
 15. The method of claim 9, further comprising:identifying a particular piece of content from a content listing;determining whether a timeline exists for the particular piece ofcontent; and generating the timeline for the particular piece of contentif it is determined that a timeline does not exist.
 16. The method ofclaim 15, further comprising adding supplemental content to the timelinebased on results of at least one of an audio recognition process and animage recognition process.
 17. A method, comprising: presenting a screenon a user device, the screen including a timeline that corresponds to aduration of a piece of content; modifying the timeline to add a firstmarker to trigger a first piece of supplemental content to be presentedon the screen in response to an instruction received at the user devicefrom an administrative entity; and modifying the timeline to add asecond marker to trigger a second piece of supplemental content to bepresented on the screen in response to a user input provided through theuser device.
 18. The method of claim 17, further comprising: presentingdifferent pieces of supplemental content in the timeline on the screenin synchronization with primary content presented on a separate device,wherein the supplemental content and the primary content are associatedwith one another; and presenting different pieces of supplementalcontent on the screen in response to selections, by a user of the userdevice, of different portions along the timeline.
 19. The method ofclaim 17, wherein the user input provided through the user device is aselection of a share option to share entered data with other users. 20.The method of claim 17, further comprising: receiving one or moremessages generated during a presentation of content on a separatedevice; and identifying points along the timeline corresponding topoints in time during presentation of the content when the one or moremessages were generated.